nach oben

Erschienen in:

2024 | OriginalPaper | Buchkapitel

AI-Based Decision-Making and the Human Oversight Requirement Under the AI Act

verfasst von : Claes G. Granmar

Erschienen in: YSEC Yearbook of Socio-Economic Constitutions 2023

Verlag: Springer Nature Switzerland

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

In the EU, software classified as artificial intelligence (AI) is considered to categorically pose a high risk to human health, safety and fundamental rights if it is applied in relation to products covered by the system for public surveillance and enforcement known as the “new legal framework”, and when implemented in certain areas beyond the general product safety regime. Whereas private parties carry the enforcement costs regarding low- or no-risk AI, the EU Member States must, according to the AI Act, have a dialogue with those who provide and apply high-risk AI professionally to ensure compliance prior to marketing and as long as the system is in use. Since the AI system learns and adapts, Article 14 of the AI Act provides that a natural person shall be designated by the professional user to oversee the high-risk AI system continuously. However, the AI Act does not define the rights and interests that it is intended to safeguard. A case in point is the right to data protection, which is mainly particularised in the General Data Protection Regulation (GDPR). Measures typically need to be taken ex ante to ensure that data processing by a high-risk AI system complies with the data subject’s right under Article 22 of the GDPR “not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her”. In that context, the relationship between the system for impact assessment established primarily by Article 35 of the GDPR and the compliance regime introduced by the AI Act is explored.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Everybody Wants To Rule the World: The Relevance of the Rule of Law for Private Law in the Context of Algorithmic Profiling of Online Users

Nächstes Kapitel Algorithmic Hiring Systems: Implications and Recommendations for Organisations and Policymakers

Nur mit Berechtigung zugänglich

See Communication from the European Commission, Artificial Intelligence for Europe, COM(2018) 237 final, 25.4.2018. See also, Boden (2018).

See e.g., Diakopoulos (2019). See as to generative AI e.g., Miller (2020). See decisions of the European Patent Office (“EPO”) of 27 January 2020 on application 18 275 163 and 18 275 174. See also in UK law, Thaler (Appellant) v. Comptroller-General of Patents Designs and Trademarks (Defendant) UKSC 2021/0201 [2021] EWCA Civ. 1829. See also [2022] Bus LR 375, [2021] EWCA Civ. 1374, [2021] RPC 19.

See e.g. Boström (2016); Tegmark (2018); Future of Life Institute Open Letter 2023 https://futureoflife.org/open-letter/pause-giant-ai-experiments/ last visited 2023-12-05; UK Government AI Safety Institute www.gov.uk/government/news/prime-minister-launches-new-ai-safety-institute, last visited 2023-12-05; and President Bidens Executive Order on Safe, Secure and Trustworthy AI, www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/ last visited 2023-12-05.

See as to labour law e.g. De Stefano, V. (2018), International Labour Organisation Working Paper No. 246, Negotiating the algorithm: Artificial Intelligence and labour protection; and Aloisi and De Stefano (2022).

Compare with Hayek (2007) advocating unrestrained liberalism as a way out of central planning to freedom.

See the European citizen’s proposals 35(3), 35(8) and 37(3) in the report from the conference on the future of Europe https://conference-delegation.europarl.europa.eu/cmsdata/267078/Report_EN.pdf, last visited 2023-12-05.

Consolidated version of the Treaty on European Union and Consolidated version of the Treaty on the functioning of European Union, Official Journal (“OJ”) C 326/1, 26.10.2012; and Charter of the Fundamental Rights of the European Union, OJ C 326/391, 26.10.2012.

European Declaration on Digital Rights and Principles for the Digital Decade, 2022 Users/Admin/Downloads/European_Declaration_on_Digital_Rights_and_Principles_signed_002_sPqb0eme38LoyyLQatnQJjkCZGU_94370.pdf, last visited 2023-12-05.

A provisional agreement on the text of the AI Act was reached on the 8^th of December 2023. See for the time being Commission’s Proposal for a Regulation of the European Parliament and the Council laying down harmonised rules on artificial intelligence and amending certain Union legislative acts, COM(2021) 206 final, 2021/0106 (COD), 2021.04.21. See the European Parliament proposed amendments to the European Commission’s proposal, P9_TA(2023)0236, 14.06.2023.

See as to prohibited AI applications, Article 5 of the Commission proposal for an AI Act, supra note 9, and as to transparency requirements in Article 52 of the AI Act.

See also recitals 33, 43, 48, 71 and 72 in the preamble of the Commission Proposal for an AI Act, supra note 9.

Article 16 of the TFEU on the right to data protection is highlighted as the legal basis for the AI Act in the preamble of the Commission’s Proposal for an AI Act, supra note 9.

Regulation (EU) 2016/697, General Data Protection Regulation, OJ L 119/1, 04.05.2016. See also Regulation (EU) 2018/1725, EU Institutions and Data Protection Regulation, OJ L 295/39, 21.11.2018; and Directive (EU) 2016/680, on data protection and law enforcement, OJ L 119/89, 04.05.2016.

Bygrave (2020), pp. 522–542.

Judgement of 7 December 2023, OQ v Land Hessen (“SCHUFA Holding I”), Case C-634/21, EU:C:2023:957.

See as to the possibility for national authorities to interpret and apply the GDPR when enforcing legal regimes other than the GDPR, judgement of 4 July 2023, Meta Platforms and Others, C-252/21, EU:C:2023:537. See as to the data subject’s status as a “consumer”, Judgement of 25 January 2018, Maximillian Schrems v Facebook Ireland Limited, Case C-498/16, EU:C:2018:37. See as to the possibility for consumer organisations to invoke the GDPR on behalf of the data subjects under Article 80 of the GDPR, Judgement of 28 April 2022, Meta Platforms Germany, Case C-319/20, EU:C:2022:322.

Compare Article 3(1) in the Commission’s Proposal for an AI Act, and the amendments proposed by the Parliament, supra note 9. See section 1 of the OECD, Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449.

See as to the legislative procedure Articles 4, 7(1), 11(3), 43(5) and (6), 48(5), 73 and 74 of the Commission’s Proposal for an AI Act, supra note 9. See also Article 291 in the TFEU, and Regulation (EU) No. 182/2011 on the possibility for the Member States to control the Commission’s exercise of its implementing powers, OJ L 55/13, 28.02.2011.

A “general purpose AI system” such as ChatGTP4 does not in itself entail a high risk, but as it may easily become a systematic threat these systems are subject to transparency requirements under Article 52 of the AI Act. See the Future of Life Institute, General Purpose AI and the AI Act, https://artificialintelligenceact.eu/wp-content/uploads/2022/05/General-Purpose-AI-and-the-AI-Act.pdf, last visited 2023-12-14. See also Lucci (2023), pp. 1–23. See also the AI Act the amendments proposed by the Parliament regarding Article 52 of the proposed AI Act, supra note 9.

According to Article 30 of the Commission’s Proposal for an AI Act, supra note 9, each Member State shall designate or establish a notifying authority responsible for organising the work of the AI conformity assessment bodies pursuant to Articles 31 - 39 of the proposed Act. See also Regulation (EC) No 765/2008 in accordance with EN-ISO/IEC 17065/2012, as to the NLF supra note 3.

See e.g., Directive 2006/42/EC, on machinery, OJ L 157/24, 09.06.2006, and European Commission, Proposal for a Regulation of the European Parliament and the Council on machinery products, COM/2021/202 final, 21.04.21.

See as to horizontal NLF framework applying to several specific acts listed in Annex II, Regulation (EC) 765/2008 on market surveillance, OJ L 218/30, 13.08.2008; Decision 768/2008/EC on market surveillance and compliance of products and amending several legislative acts, OJ L 218/82, 13.08.2008; and Regulation (EU) 2019/1020, on market surveillance and compliance of products amending several legislative acts, OJ L 169/1, 25.06.2019.

See as to conformity assessment, CE marking, documentation, and registration of high-risk AI systems, Articles 43 - 50 of the Commission’s Proposal for an AI Act, supra note 9. See as to monitoring and compliance, Articles 61 - 68 of the proposed AI Act including the provisions in Articles 63 and 65 thereof complementing Regulation (EU) 2019/1020, supra note 22.

For instance, the EU database for stand-alone high-risk AI systems established by Article 60 of the AI Act requires a definition of “safety component”.

See Article 6(1) and Appendix II of the Commission’s Proposal for an AI Act, supra note 9 Appendix III of the Commission’s Proposal for an AI Act, supra note 9, specifies the areas of activities referred to in Article 6(2) of the proposed AI Act.

See primarily Articles 5(2), 3, 2, 6(1) of the TEU in that order and Articles 51 and 52 of the EU Charter, supra note 7. See recently e.g., the European Commission’s Proposal for a Directive on Corporate Sustainability Due Diligence amending Directive (EU) 2019/1937, COM/2022/71 Final 2022/0051(COD) 2023.02.2022.

See as to the concept of a “methodological source code” of EU law in Granmar (2021), pp. 225–244, at 227 et seq. See as to the implication of the duty of sincere cooperation, judgement in Meta Platforms and Others, C-252/21, supra note 16, paras. 53 and 54. See also judgement of 7 November 2013, UPC Netherlands, C-518/11, EU:C:2013:709, para. 59 and judgement of 1 August 2022, Sea Watch, C-14/21 and 15/21, EU:C:2022:604, para. 156.

Judgement of 26 February 2013, Åklagaren v H. Å. F., C-617/10, EU:C:2013:105, para. 21.

See as to proportionality Article 52 of the EU Charter, supra note 7.

See Article 263 of the TFEU and e.g., Judgement of 21 December 2016, Tele2 Sverige AB and Secretary of State for the Home Department, joined Cases C-203/15 and C-298/15, EU:C:2016:970.

See for instance the interrelation between data protection and freedom of expression that applies also to AI systems, Judgement of 6 November 2003, Lindquist, C-101/01, EU:C:2003:596, paras. 89–90.

Compare with Commission report by Crémer, J., de Montjoye, Y-A, and Schweitzer, H., Competition Policy for the Digital Era, (2019) https://ec.europa.eu/competition/publications/reports/kd0419345enn.pdf, last visited 2023-12-13.

Hence, Article 114 of the TFEU on the internal market is highlighted as the legal basis for the AI Act in the preamble of the Commission’s Proposal for an AI Act, supra note 9.

See the World Forum for the harmonisation of vehicle regulations (UNECE WP 29) unece.org/transport/vehicle-regulations, last visited 2023-12-13. See also Mc Fadden et al. (2021); and Ryck et al. (2020), p. 152.

See judgement of 27 October 2016, James Elliot Limited, C-613/14, EU:C:2016:821. It may be considered problematic that legal acts with the same effect as legislation are adopted without a legislative process.

See also Article 42 of the Commission’s Proposal for an AI Act, supra note 9.

Article 41 of the Commission’s Proposal for an AI Act, supra note 9.

See, the GDPR, supra note 13. See as to cybersecurity Article 15 of the Commission’s Proposal for an AI Act, supra note 9, and Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union amending Regulation (EU) 910/2014 and Directive (EU) 2018/1972, OJ L 333/80, 27/12/2022 (“NIS 2 Directive”). See also Regulation (EU) 2019/881 on ENISA and on information and communication technology cybersecurity certification, OJ L 151/15, 07/06/2019 (“Cyber Security Act”).

See also the European Commission’s White Paper on Artificial Intelligence, supra note 1.

See as to the data minimisation requirement under Article 5(1)(c) of the GDPR in Opinion by AG Pikamäe of 16 March 2023, UF and AB v Land Hessen (“SCHUFA Holding II”), joined Cases C-26/22 and C-64/22, EU:C:2023:222, para. 87.

See Article 59 of the Commission’s Proposal for an AI Act, and amendments proposed by the Parliament, supra note 9.

See Articles 51–66 of the GDPR, supra note 13, and the duty of national competition authorities to cooperate with DPA:s, judgement in Meta Platforms and Others, C-252/21, supra note 16, paras. 56 to 61.

See Articles 3(1)(4) and 16-29 of the Commission’s Proposal for an AI Act, supra note 9.

See e.g., Perloff (2018). It cannot be ruled out that product development can constitute a legitimate interest that needs to be reconciled with other legitimate interests such as data protection, see judgement in Meta Platforms and Others, C-252/21, supra note 16, para. 122. See Articles 53 - 55 of the Commission’s Proposal for an AI Act.

See Article 52 of the Commission’s Proposal for an AI Act, and amendments proposed by the Parliament, supra note 9.

Compare with Article 12 of the GDPR, supra note 13.

See the Commission’s Proposal for an AI Act, and amendments proposed by the Parliament, supra note 9.

Software will soon be integrated with humans, https://www.neuralink.com, last visited 2023-12-05. See as to the concept of being “human” European resolution of 16 February 2017 with recommendations to the Commission on Civil Law Rules on Robotics (2015/2103/(INL)), P8_TA(2017)0051.

See as to generic AI that invents and creates new things, supra note 2.

See also the limitations in Article 2(2) of the Commission’s Proposal for an AI Act, supra note 9.

Compare with the notion of “controller” in Article 4(7) of the GDPR, supra note 13.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, paras. 52 and 64; and Opinion of Advocate General Pikamäe of 16 March 2023 on the SCHUFA Holding I, Case C-634/21, EU:C:2023:220, para. 31.

See further as to the notion of controller, Judgement of 13 May 2014, Google Spain SL and Google Inc. v Agencia España Proteccion de datos (AEPD) and Mario Costejo González, Case C-131/12, EU:C:2014:317.

Ibid.

See Kingwell (2020).

Opinion of Advocate General Pikamäe of 16 March 2023, Case C-634/21, supra note 52.

See Articles 17 and 11 of the EU Charter.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 43.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 45. See the requirement to use appropriate mathematical or statistical procedures for profiling in recital 71 of the preamble to the GDPR, supra note 13. See also Opinion of the Advocate General in SCHUFA Holding I, Case C-634/21, supra note 52, paras. 37, 38, and 43.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 44.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 46.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 47.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 48.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 49. See also Opinion of the Advocate General in SCHUFA Holding I, Case C-634/21, supra note 52, para. 35.

Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, paras. 61 and 73.

Opinion of the Advocate General in SCHUFA Holding I, supra note 52, para. 44.

Opinion of the Advocate General in SCHUFA Holding I, supra note 52, para. 54. See also Article 15(1)(h) of the GDPR, and recitals 58 and 63 in the preamble to the GDPR, supra note 13.

Opinion of the Advocate General in SCHUFA Holding I, supra note 52, para. 50. In fact, this line of reasoning is based on observations submitted by the referring court referred to by the ECJ, see Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 23.

Recital 58 in the preamble to the GDPR, supra note 13. See also judgement in Meta Platforms and Others, C-252/21, supra note 16, para. 122 where the ECJ recognises that product development can justify limitation in data protection.

Compare with judgement of 10 July 2018, Tietosuojavaltuutettu, Case C-25/17, EU:C:2018:551.

Judgement of 7 December 2023, UF and AB v Land Hessen (SCHUFA Holding II), Joined Cases C-26/22 and C-64/22, EU:C:2023:958.

See also Opinion of the Advocate General in SCHUFA Holding I, supra note 52, para. 61.

judgement in Meta Platforms and Others, C-252/21, supra note 16, paras. 140–154 addressing the definition of consent enshrined in Article 4(11) of the GDPR, supra note 13.

Judgement in Meta Platforms and Others, C-252/21, supra note 16, paras 97–98.

See Judgement in SCHUFA Holding I, Case C-634/21, supra note 15, para. 55.

Opinion of the Advocate General in SCHUFA Holding I, supra note 52, para 49.

See recital 43 in the preamble to the GDPR, supra note 13, and judgement in Meta Platforms and Others, C-252/21, supra note 16, para. 149.

Compare Article 2(2) of the Commission’s Proposal for an AI Act with Annex III, p. 8, thereof, supra note 9.

Ibid. Compare with Susskind (2019).

Article 80 of the GDPR, supra note 13. See also Judgement in Maximillian Schrems v Facebook Ireland Limited, Case C-498/16, supra note 16; and Judgement in Meta Platforms, Case C-319/20, supra note 16. See also, Jain (2022).

If the software concerned in the SCHUFA Holding I Case belonged to the AI technology family, it was a high-risk AI system pursuant to point 5(b), Annex III, of the Commission’s Proposal for an AI Act, supra note 9.

Article 37 of the GDPR, supra note 13.

Article 43 and point 5(b), Annex III, of the Commission’s Proposal for an AI Act, supra note 9, and Regulation (EC) No 765/2008 in accordance with EN-ISO/IEC 17065/2012, see further as to the NLF supra note 22.

Article 43(7) of the Commission’s Proposal for an AI Act, supra note 9.

See as to the definition of market surveillance authority Article 3(28) of the Commission’s Proposal for an AI Act, supra note 9, and Articles 63–68 thereof, aligning the system for market surveillance and enforcement under Regulation (EU) 2019/1020, supra note 22.

See also Article 61 of the Commission’s Proposal for an AI Act, supra note 9, requiring providers to establish a post-market monitor system that is proportionate to the nature of the technology and the duty to report serious incidents under Article 62 thereof.

Article 65(1) of the Commission’s Proposal for an AI Act, supra note 9.

Compare with the duty of national competition authorities to cooperate with DPA:s, judgement in Meta Platforms and Others, C-252/21, supra note 16, paras. 37–63.

Judgement in Meta Platforms and Others, C-252/21, supra note 16, paras. 43–44.

Judgement in Meta Platforms and Others, C-252/21, supra note 16, para. 48.

See Commission’s Proposal for an AI Act, supra note 9, chapter 1.1.

See Article 53 in the Commission’s Proposal for an AI Act, supra note 9.

For those who are interested in theology, the story in the Jewish Christian tradition is found in Genesis 2–3.

See as to the error in the logic of making “efficiency” an objective, Dworkin (1980). p. 1 et seq.

Quote from Lee (2019), p. 5. See, for an overview, also Lee (2018); Zuboff (1989); Winner (1986); Russel and Norvig (2016); M. Tegmark, Life 3.0, supra note 3; and N. Bostrom, Superintelligence, supra note 3.

Holy Bible, Exodus 32.

See for an overview of the development of the provision in Bygrave (2020), supra note 14; and the EU Declaration on Digital Rights, supra note 8.

See European Commission’s Proposal for a Directive on adapting non-contractual civil liability rules to artificial intelligence, COM/2022/496 final (AI Liability Directive).

Aloisi A, De Stefano V (2022) Your boss is an algorithm. Hart PublishingCrossRef

Boden MA (2018) Artificial intelligence: a very short introduction. Oxford University PressCrossRef

Boström N (2016) Superintelligence. Oxford University Press

Diakopoulos N (2019) Automating the news: how algorithms are rewriting the media. Harvard University PressCrossRef

Hayek FA (reprint 2007) The road to serfdom. University of Chicago Press (originally 1944 Routledge Press)

Kingwell MA (2020) In: Dubber MD et al (eds) Sentient AIs Persons? in the Oxford Handbook on Ethics of AI. Oxford Academic

Lee K-F (2018) AI Superpowers, China, Silicon Valley, and the New World Order. Houghton Mifflin Harcourt

Miller AI (2020) The artist in the machine: the world of AI powered creativity. Massachusetts Institute of Technology Press

Perloff, J. M. (2018) Microeconomics, Global edition, 8th edn. Pearson

Russel S, Norvig P (2016) Artificial intelligence: a modern approach, 3rd edn. Pearson

Tegmark M (2018) Life – being human in age of artificial intelligence. Penguin Books

Winner L (1986) The whale and the reactor. University of Chicago Press

Zuboff S (1989) In the age of the smart machines: the future of work and power. Basic Books

Bygrave LA (2020) Article 22 Automated individual decision making, including profiling. In: Kuner C, Bygrave LA, Docksay CD, L. (eds) The EU general data protection regulation (GDPR): a commentary. Oxford University Press, pp 522–542CrossRef

Dworkin R (1980) Why efficiency? – a response to professors Calabresi and Posner. Hofstra Law Rev 8(5), article 5:1

Granmar CG (2021) Global applicability of the GDPR in context. Int Data Priv Law 11(3):225–244CrossRef

Jain P (2022) Standing for Non-Profit Associations or Consumer Associations to File Collective Actions without a Data-Subjects Mandate: Case C-319/20 Meta Platforms. European Law Blog, 16 May 2022

Lee K-F (2019) Youth Law Journal (YJL), 2nd edn. Oxford

Lucci N (2023) ChatGTP, a case study on copyright challenges for generative artificial intelligence systems. Eur J Risk Regul 1–23

Mc Fadden M, Jones K, Taylor E, Osborn G (2021) Harmonising artificial intelligence: the role of standards in the EU AI regulation. Oxford Commission on AI & Good Governance

Ryck MD, Verstayhe M, Debrouwere D (2020) Automated guided vehicle systems, state-of-the-art control algorithms and techniques. J Manuf Syst 54:152CrossRef

Titel: AI-Based Decision-Making and the Human Oversight Requirement Under the AI Act
verfasst von: Claes G. Granmar
Verlag: Springer Nature Switzerland
Buch: YSEC Yearbook of Socio-Economic Constitutions 2023
Print ISBN: 978-3-031-55831-3

Electronic ISBN: 978-3-031-55832-0

Copyright-Jahr: 2024
DOI: https://doi.org/10.1007/16495_2024_68

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"