Amazon Comprehend

Executive Summary

Amazon Comprehend is AWS's managed language-analysis service. It offers predefined analysis such as sentiment and entity detection, plus custom classifiers trained on a customer's labeled documents. The latter is its relevant decision-API surface: software submits text or a document and receives category scores, then applies its own routing, review, or business policy. ^[1][2]

The strongest reason to evaluate it is a repeatable document-labeling problem with training data and an existing AWS operating environment. Important tradeoffs include dataset preparation, endpoint capacity costs, document limits, and the distinction between model scores and measured decision accuracy. This review checked public documentation on September 16, 2026; it did not train a classifier or benchmark the service. Compare these tradeoffs in the AI decision APIs comparison.

Availability requires a feature-level check. Since April 30, 2026, topic modeling, event detection, and prompt safety classification are unavailable to new customers. AWS says accounts that used those features during the preceding 12 months retain access, and other Comprehend features are unaffected. An old tutorial or pricing entry for prompt safety is insufficient evidence that a new account can use it. ^[6]

---

Product Overview

Two kinds of classification

Built-in models answer predefined language-analysis questions. Custom classification instead learns the organization's categories—such as invoice type or support queue—from examples. It is not a prompt-only rubric evaluator: the custom workflow includes data preparation and training before inference. ^[1][2][3]

The label mode matters. Multi-class training expects one mutually exclusive class per document; multi-label training permits several categories on the same document. In the API, these produce Classes and Labels, respectively. A downstream system must handle that distinction instead of assuming every response is one winning label. ^[3][8]

---

Training and Evaluation

A practical starting workflow is to define a stable label taxonomy, prepare examples in a supported format, select the document model and language, train, inspect held-out performance, and then deploy or run batch inference. For CSV training, AWS lists a minimum of 50 documents per class for multi-class models. Multi-label CSV training requires at least 10 per class and 50 documents overall. Those are service minima, not evidence that a small dataset represents production traffic. ^[3][9]

By default, Comprehend uses 90% of the supplied data for training and 10% for testing. A caller can provide a separate test set. AWS warns that rare labels may be absent from an automatic test split; an explicitly supplied test set must cover the training labels. A careful evaluation should also separate near-duplicate documents and distinguish later traffic from data already seen during development. The last two checks are evaluation recommendations, not automatic Comprehend guarantees. ^[10]

The service reports metrics including accuracy, precision, recall, and F1, with macro/micro measures and multi-label-specific measures where applicable. These answer different questions from calibration. A strong aggregate F1 does not establish that predictions scored 0.9 are correct 90% of the time, or that a rare, expensive error is acceptable. ^[11]

AWS defines a returned class Score as the model's confidence that the class is correctly assigned. The reviewed API documentation does not establish workload-independent calibration. Set thresholds using adjudicated examples from the intended workload; evaluate accepted-case error rates and the fraction sent for human review. ^[12]

---

Runtime and Deployment

Calling a trained classifier

This illustrative AWS CLI request assumes an existing trained model, a running endpoint, configured credentials, and permission to call it. Replace the endpoint environment variable with the appropriate ARN. It was checked against the request schema, not executed against a paid endpoint. ^[8]

aws comprehend classify-document \
  --endpoint-arn "$COMPREHEND_ENDPOINT_ARN" \
  --text 'The invoice lists a duplicate charge for order 104.'

The response supplies class or label scores. When sending file bytes, it can also contain extracted-document metadata, page-level errors, and warnings. A successful HTTP response alone therefore does not establish that every supplied page was classified cleanly. Validate error/warning fields before turning results into actions. ^[8]

Plain-text synchronous inputs are limited to 10 KB. For supported semi-structured synchronous inputs, the documentation specifies one page with at most 10,000 characters, plus format-specific byte limits. Route longer collections through the appropriate asynchronous workflow or make an explicit splitting policy; silently truncating a document changes the decision problem. ^[9]

An inference unit provides capacity for 100 characters per second, subject to a maximum of two documents per second per unit. This is a throughput specification, not a per-request latency promise. Size endpoints using document length and peak arrivals, then measure tail latency and backpressure under realistic load. Endpoint capacity and the attached model version can be updated. ^[9][5]

Updating models deliberately

Named versions record changes in training data and evaluation results. Training a new version requires the full relevant dataset, including previous and new documents. Updating the serving endpoint is a separate operation; the existence of a newly trained model does not itself prove production switched to it. ^[4][5]

Flywheels organize iterative training for plain-text custom classifiers and entity recognizers. They collect training/test datasets in an S3 data lake, evaluate a current model, train and evaluate a candidate, and leave selection of the active model to the operator. They do not remove the need to review quality or release changes. ^[13][14]

Two limits are operationally useful. A flywheel's label set is fixed at creation; labels outside that set in later datasets are ignored. Deleting a flywheel also leaves its data-lake files intact, so resource deletion and data deletion require separate attention. These are flywheel-specific constraints, not a claim that the broader model-versioning workflow can never change labels. ^[13][4]

---

Data and Security Boundaries

The Comprehend FAQ says AWS may store and use general-service inputs to improve its AI services. It also describes possible storage outside the selected region for that purpose. The FAQ carves out Comprehend Medical and Detect PII from that particular improvement-use answer; those exceptions must not be generalized to a custom classifier. ^[19]

AWS Organizations provides opt-out policies for covered AI services. The policy documentation says opting out deletes historical content stored for improvement, while retaining content needed for service functionality. Confirm the account's effective policy and the actual data lifecycle; neither private connectivity nor improvement opt-out is a zero-data-retention claim. ^[18]

---

Pricing and Operating Cost

Published custom-classification rates checked September 16, 2026: ^[20]

Endpoint charges continue until deletion, including idle time. At that rate, one IU is $1.80/hour, or $1,296 for 30 continuous days. These are arithmetic examples, excluding training, model management, document extraction, storage, and other associated services. Confirm regional pricing and the required throughput before budgeting. ^[20]

For illustration, 1,000 documents of 300 characters each cost $1.50 in async inference at the stated unit rate. The same sparse workload can have a very different cost structure if it requires an endpoint available around the clock. Compare batch delay, peak capacity, and accepted-case quality—not just a nominal per-document price. ^[20]

Licensing model: paid managed service. This review did not identify a downloadable custom-model package for operating the trained classifier outside Comprehend; AWS network controls should not be described as self-hosting.

---

What Developers Say

In a February 2021 r/aws discussion, Aggressive_Area2522 described evaluating several classifiers for a system that needed synchronous classification available 24/7. Their concern was the cost of keeping endpoints running despite modest traffic. The thread's historical cross-provider price comparison is not a current benchmark; the underlying idle-capacity concern remains consistent with today's AWS billing documentation. ^[21][20]

In September 2024, NoSeat2946 described a law firm's document-sorting problem: roughly 130 form types, with new forms arriving each year, made retraining and splitting classifiers difficult to plan. This is a self-reported workflow constraint, not proof of general accuracy or adoption. Current versioning and flywheel documentation help explain the design issue: updating a model requires training work, and a flywheel does not automatically expand its original taxonomy. ^[22][4][13]

These dated reports provide useful operating questions. Neither substitutes for a contemporary latency, accuracy, or calibration evaluation on the reader's documents.

---

Strengths and Cautions

Strengths

• Customer-defined labels: a managed training path for recurring document categories rather than only a fixed sentiment vocabulary. ^[2]
• Document-aware input: native-document models account for text and layout in supported English files. ^[7]
• Explicit releases: named versions, evaluation metrics, and controlled endpoint updates make model changes visible to the operator. ^[4][11][5]
• AWS deployment controls: private API networking and configurable job networking/encryption fit an existing AWS operating model. ^[15][16][17]

Cautions

• Training is part of the product: a new label taxonomy needs examples and evaluation; this is not a zero-shot prompt endpoint. ^[3]
• Always-on costs matter: low utilization can dominate the economics of synchronous classification. ^[20]
• Scores need validation: the API's confidence field is not an independent accuracy guarantee. ^[12]
• Availability is uneven: do not build a new-account plan around the three restricted features. ^[6]

---

Alternatives and Execution Context

TypeSafe/Jev addresses a different starting point: define Choice, Score, or Noul questions over application context and compose the returned values in code. It is relevant when testing judgments without first creating a document-label training corpus. Comprehend is relevant when a learned, versioned document taxonomy is the actual requirement. That is a workflow distinction, not an accuracy ranking. ^[23][3][4]

Tembo is complementary execution infrastructure. Its current platform runs coding agents with repository/ticket context and produces work for review. In an application-designed workflow, a classifier could help route a support report before approved engineering work is handed to an agent platform. Comprehend supplies a label; application policy decides whether work proceeds; Tembo runs the selected agent task. No native Comprehend–Tembo integration was verified. Disclosure: Ry Walker is Tembo's CEO and co-founder. ^[24]

Assessment

Comprehend is worth evaluating for teams with labeled text or document corpora, reasonably stable categories, and the ability to operate AWS training and serving resources. Batch-friendly workloads avoid assuming that every classification needs an always-on endpoint. A team with no labels, frequently changing prompt-defined rubrics, or a requirement to run model weights locally should first examine other approaches. These are fit judgments based on the workflow and boundaries above.

The current custom-classification documentation supports continued evaluation; the narrower April availability change is a reason to verify the exact feature before committing, not evidence that the entire service has shut down. Future product longevity and performance on a particular corpus remain unproven by this documentation review. ^[6][2]

---

Research by Ry Walker Research • methodology