AI agents in the German Mittelstand: a practical guide to selection, integration, and operation

AI agents are software systems that understand a goal, draw on company data, and use approved tools such as APIs, databases, or Microsoft 365 services. In mid-sized companies they pay off above all for frequent, clearly delimited workflows: finding technical knowledge, preparing service requests, checking documents, consolidating data from several systems, or handling recurring cases.

The sensible starting point is not an autonomous 'super-agent'. It is a tightly scoped agent with a measurable result, unambiguous data access, and human approval for critical actions. Only once quality, security, and cost-effectiveness are proven in the real process is the scope expanded.

In 2025, according to the German Federal Statistical Office, 26 percent of German companies with ten or more employees used AI. Among companies with 50 to 249 employees the share was 36 percent. For automating workflows or decisions, 27 percent of AI-using companies deployed the relevant technologies. The figures show growing adoption, but say nothing yet about the quality or productivity of individual implementations. Source: German Federal Statistical Office

That is exactly where the real work begins: a chat window is set up quickly. An AI system that understands technical documents, respects user permissions, operates approved company systems, and runs reliably day to day is an integration and operations project.

Microsoft describes agents as AI systems that interpret data, support decisions, and automate tasks. They can work in a purely conversational way or execute workflows with a certain degree of autonomy. Microsoft Learn

A modern AI agent typically has five components: The model processes language and unstructured information. The instructions define the goal, behavior, and boundaries. The context supplies relevant company knowledge. The tools connect the agent to search systems, databases, and APIs. The control layer checks permissions, inputs, actions, and results.

The language model is therefore only one component. Whether an agent is usable in production is decided by the application as a whole: which data may it read? Which actions may it perform? How is an error detected? Who approves critical steps? How can a case be traced later?

A classic chatbot answers questions. It often draws only on general model knowledge or a narrowly limited body of information.

A knowledge assistant augments the model with internal sources. It can, for example, search technical documentation, process descriptions, or a quality-management manual and back up its answer with references. It normally does not change any data in the source systems.

A classic automation executes predefined steps. When a particular event occurs, a defined action is performed. For stable, fully describable processes this is often the best solution.

An AI agent can decide, within given boundaries, which approved tool is needed next. It can, for example, classify an email, look up a customer case in the CRM, retrieve matching product data, and then prepare a draft reply.

The transitions are fluid. What matters is not the label, but the actual scope of functionality.

Mid-sized companies usually already have the most important starting points: ERP systems, CRM, Microsoft 365, SharePoint, document management, and large amounts of process knowledge. At the same time, skilled staff are scarce, knowledge is concentrated in a few experienced people, and many workflows involve repetitive searching, checking, or transferring of data.

Mid-sized companies operate under different conditions than large corporations. There are fewer specialists for data platforms, AI security, and model operations. Projects have to show concrete value faster and must not tie up several internal teams permanently.

The OECD names missing skills, limited financing, scarce resources, and insufficient digital infrastructure as recurring obstacles to AI adoption in small and medium-sized enterprises. It also distinguishes between companies that use individual off-the-shelf tools and those that deliberately integrate AI into their core processes. OECD: AI adoption by small and medium-sized enterprises

In our project experience, the model is rarely the first bottleneck. Four other questions are more critical: Are the required data findable and sufficiently maintained? Does the source system have a usable interface? Can user and system permissions be mapped cleanly? Is there someone responsible for judging domain quality and exceptions?

A service technician stands in front of a machine and searches for a specific error message. The information is spread across manuals, old service reports, SharePoint folders, and the knowledge of experienced colleagues.

A knowledge agent can search these sources and return an answer with references. This is especially valuable for model numbers, error codes, spare parts, and concrete work steps. Purely semantic search is often not enough for this. In practice, a combination of semantic search, classic full-text search, and a subsequent ranking of the hits has proven effective.

In a production Hoch-AI project, SharePoint serves as the maintained knowledge source and Microsoft Teams as the interface for the technicians. Changes to the approved documents are regularly picked up into the search index. The most important lesson was not the choice of language model: document structure, synchronization, permissions, and source citations determined whether the answers were trustworthy in everyday work.

A public example is the Teams knowledge bot for field service and support.

An email agent can classify incoming messages, find the relevant cases, and prepare a reply. In customer service it can also check whether information is missing and route the case to the right department.

A sensible first level of maturity reads and structures. It does not send every message on its own. Employees see the draft, the information used, and the proposed action.

Only once classification and answer quality have been measured across enough real cases can clearly defined cases be automated further. Complaints, legally relevant statements, payments, or unusual business transactions remain in the human decision path.

Incoming invoices, delivery notes, order confirmations, and forms contain structured information in unstructured form. A document agent can recognize document types, extract relevant fields, and reconcile them with order or master data.

The actual agentic part begins where context is needed: which order does this document belong to? Is a discrepancy plausible? Which information is missing? Does an employee need to check?

The agent should flag exceptions and only post or change data in a system of record according to defined rules. High extraction accuracy alone is not enough. What matters is how errors, duplicates, and contradictory entries are handled.

In mechanical engineering and manufacturing, quality reports are assembled from many individual pieces of information: error description, measured values, photos, affected components, previous incidents, and implemented measures.

A quality agent can guide the process, search historical reports for similar failure patterns, and draft wording for individual sections. Measured values and causes must not be invented or derived from a plausible-sounding answer. They must come from approved sources.

The responsible quality employee remains the owner of the assessment. Root-cause analysis and the approval of measures in particular are not merely a writing task.

In technical sales, an inquiry often has to be reconciled with product data, availability, previous quotes, and technical constraints. When this information is spread across several systems, a lot of searching and transferring is required.

An agent can structure the requirements from an inquiry, search for matching products or components, and prepare a draft quote. Prices, discounts, delivery commitments, and special technical approvals should still come from authoritative systems and rules.

This use case shows particularly clearly: without reliable product and master data there is no reliable agent. Sometimes building a central data foundation is therefore the necessary first project step.

A reporting agent can consolidate data from approved sources, describe deviations, and answer follow-up questions in natural language. Possible areas include production metrics, service cases, sales data, or project status.

The figures themselves must come from the system of record. The language model helps with interpretation and presentation, but does not replace the defined metric logic.

A good reporting agent therefore indicates which period and which source a figure comes from. It should also distinguish between a measured value, a derived metric, and a linguistic interpretation.

An AI agent does not have to appear as a separate chat tool. It can be integrated directly into an existing customer portal, a line-of-business application, or a SaaS product.

In such a setup, the existing application authenticates the user. The agent receives the tenant and user context server-side and can only call the API functions approved for it.

Planning a multi-tenant agent backend yielded an important lesson: tenant, roles, and approved tools must not be taken from freely modifiable client input or from the prompt. These boundaries must be enforced technically in the backend. A prompt is not a security boundary.

An AI agent is not automatically better than classic software. It is usually the wrong choice when the process can be fully described with fixed rules, only a few cases occur per month, the required data is not maintained or not accessible, the source system has no robust interface, there is no domain-responsible process owner, a single error would immediately have serious or irreversible consequences, or no baseline for time, quality, or cost is known.

Rigid approval routing belongs in a workflow engine. A simple data transfer belongs in a normal integration. A document search can first be implemented as a knowledge assistant.

The right starting point does not begin with the question of a model or framework. It begins with a concrete workflow.

Six criteria have proven useful for the selection: Frequency – the case occurs regularly and in sufficient numbers. Measurability – processing time, lead time, error rate, or follow-up queries can be compared before and after the pilot. Data access – the required information exists digitally and may be used. Integratability – the most important systems offer APIs, exports, or other stable access points. Manageable risk – errors can be detected, corrected, and if necessary rolled back. Domain responsibility – a process owner can define test cases, edge cases, and acceptance criteria.

For the first agent, one process, one team, and one clear result type are enough. A project with five departments, eight integrations, and an unclear value proposition is not a pilot, but a transformation program.

A detailed method is provided in the article Finding the right AI use case.

A production AI agent consists of several building blocks. Together they determine which data it may read, which actions it can trigger, and how errors are detected and contained. The following sections describe the most important building blocks, from the interface to evaluation.

The agent can appear in Microsoft Teams, Outlook, a web portal, or directly in a line-of-business application. It can be triggered by a user question, a new email, a document upload, or an event from a company system.

The interface should show what the agent understood, which source it uses, and which action it proposes.

Users and the agent need a unique identity. The agent may only access the data and functions required for its task.

In a Microsoft environment, existing user groups and roles can form a good foundation. Technical services should, where possible, use their own managed identities instead of static access keys.

Microsoft recommends a dedicated, auditable identity for agents, minimal permissions, and technically enforced human approvals for risky or hard-to-reverse actions. Microsoft Learn: Secure autonomous agentic AI systems

Documents, manuals, or policies are not simply copied into a prompt in full. A retrieval system first finds the relevant passages and passes only the needed context to the model.

The knowledge base needs a clearly defined source of truth. Changed and deleted documents must be reliably updated in the search index. For technical content, exact terms such as part numbers and error codes are just as important as semantically similar wording.

Anyone already using Teams, SharePoint, and Entra ID should first assess which Microsoft 365 foundation for AI is in place.

Tools are approved functions that the agent may use. Examples include: looking up a customer or order, retrieving a document status, creating a ticket, saving a draft reply, generating a report, or submitting a case for approval.

Every tool needs a defined input schema, a permission check, and controlled error behavior. The agent should not be allowed to run arbitrary database queries or freely worded commands.

Orchestration decides which steps are allowed and when a human must take over. It also manages the state of a longer-running case.

Important rules belong in application code or workflow logic, not solely in the system prompt. These include approvals, budgets, roles, permitted actions, and abort conditions.

A production agent is not tested with just a few demo questions. It needs a representative test dataset of normal cases, edge cases, and known failure patterns.

Besides answer quality, tool selection, data access, latency, cost, and error behavior are checked, among other things. The voluntary NIST AI Risk Management Framework classifies testing, evaluation, verification, and validation as part of ongoing AI risk management.

In operation, technical errors, unusual costs, failed tool calls, and quality feedback must be visible. Detailed raw content, however, does not automatically belong in every log.

Technically, the integration is usually a combination of API contracts, identities, permissions, and data models. The real difficulty lies less in calling an interface than in the domain boundaries.

A safe integration path typically looks like this: first, the systems and data flows involved are documented. Then roles and permitted actions are defined. Next, the required API functions are defined and tested one by one. The agent starts with read access or drafts. Write actions receive validation and approval steps. Errors, retries, and aborts are handled technically. Only then is the scope expanded.

The article Integrating AI agents into existing enterprise systems covers the technical prerequisites and most common hurdles in detail.

If you want to implement a concrete process in ERP, CRM, or Microsoft 365, you can find the scope of services at AI agent integration.

A pilot answers a limited question: can the agent support a specific case well enough under realistic conditions? A four-week frame can be used for a validated first solution.

Week 1 – process and baseline: the current workflow, the systems involved, and the domain exceptions are recorded. Success criteria and an initial test dataset are defined together.

Week 2 – data and first workflow: the most important data sources are connected. The agent is given a limited task and, at first, only the tools needed for it.

Week 3 – evaluation and safeguards: normal cases, edge cases, and errors are tested systematically. Permissions, logging, and approval steps are added.

Week 4 – validation with real users: a small user group works with the solution. The results are compared against the previous process and the criteria defined beforehand.

After 30 days, a validated first solution can be in place. That is not automatically a fully production-ready system. Going live and scaling depend on data access, integrations, IT approvals, governance, and operational prerequisites.

Why this distinction matters is shown in the article Why so many AI pilots never reach production.

Compared to simple chatbots, agents have a larger attack surface because they operate tools and can trigger actions in other systems. OWASP names, among other things, manipulated goals, tool misuse, excessive permissions, and poisoned context or memory as specific risks of agentic applications. OWASP Top 10 for Agentic Applications

Important safeguards are a dedicated agent identity with minimal rights, an allowlist of approved tools and actions, server-side checking of roles and tenants, validation of tool parameters and return values, separation of trusted instructions from external content, technically enforced approvals for critical actions, audit logs without unnecessary sensitive content, defined retention and deletion periods, and a tested way to disable the agent or revoke its permissions.

A more in-depth technical treatment is provided in Scaling AI applications securely with Azure.

AI agents can be operated in compliance with data protection law. EU hosting alone, however, is not full proof of this.

As soon as personal data is processed, the purpose, legal basis, data minimization, recipients, retention, data-subject rights, and technical safeguards, among other things, must be assessed. The requirements arise from the specific process and the providers used. The General Data Protection Regulation remains authoritative.

The EU AI Act is generally applicable as of 2 August 2026, with different deadlines for individual areas. The AI-literacy obligations have applied since February 2025. Whether an agent qualifies as a high-risk system is decided not by the term 'AI agent', but by its intended purpose. European Commission: AI Act and current application deadlines

Before going into production, companies should therefore clarify: what roles do the provider and operator have? Which data does the agent process? Which people can be affected? Which decisions or actions may it trigger? What human oversight is provided? How are usage, limits, and the AI interaction made transparent? Which documentation and training do the employees involved need?

This assessment does not replace legal advice.

A general price per AI agent says little. Two agents that look similar can be completely different technically.

One-time costs typically arise from process analysis, data preparation, interfaces, development, evaluation, security, and rollout. Running costs arise from hosting, model usage, databases, monitoring, maintenance, and ongoing domain development.

In practice, integration and testing effort is underestimated more often than pure model costs. A cheap model call helps little if master data is unreliable or a missing API has to be replaced with manual workarounds.

For the business case, the current process should be measured first: how many cases are there? How long does one case take? Which errors and follow-up queries occur? Which cases can the agent prepare? How many exceptions still require a human? What costs arise from a wrong decision?

A simple cost-effectiveness calculation combines the annual case volume, the realistically saved processing time, and the avoided error costs. From this, implementation and ongoing operation are subtracted. Unfounded promises of blanket savings do not belong in the calculation.

The following questions come up again and again in mid-sized-company projects.

No. An agent is only worthwhile when a concrete process benefits from language understanding, company context, or flexible tool selection. For rigid workflows, classic automation is enough.

Frequent, clearly delimited, and measurable workflows with available data are suitable. Good starting points are knowledge search, document checking, classification, draft replies, and reporting.

Usually not. AI agents are integrated into ERP, CRM, Microsoft 365, or line-of-business applications via APIs and identities. Missing or unstable interfaces can, however, increase the effort considerably.

No. Microsoft 365 is a common foundation, but not a prerequisite. Agents can also be connected to other cloud and on-premises systems. The architecture depends on the data, security requirements, and existing IT.

Technically, autonomous workflows are possible. In enterprise operations, the degree of autonomy should depend on the risk. Critical or hard-to-reverse actions require a technically enforced human approval.

A tightly scoped first solution can be validated within a few weeks. Going into production can take longer if data access, interfaces, data-protection reviews, or internal approvals are missing.

With a documented test dataset and domain criteria. Depending on the use case, answer quality, source grounding, tool selection, error rate, lead time, and user feedback are measured.

Yes, if the use case occurs frequently enough and can be implemented without disproportionate infrastructure. The solution should start with a limited process and not be planned as a comprehensive AI platform.

In mid-sized companies, AI agents can make knowledge accessible, relieve employees of repetitive searching and checking, and connect existing systems sensibly. But the value does not come from as much autonomy as possible.

A viable first agent has a tightly delimited task, a measurable baseline, reliable data sources, minimal and auditable permissions, human approval for critical actions, a representative test dataset, plus monitoring and a responsible process owner.

Whoever starts this way is not building an isolated AI pilot. They are creating a reusable foundation on which further agents and automations can be built in a controlled way. The next step is therefore not choosing a model, but examining a concrete process and its integration into your existing enterprise systems.