Moving a task from a cloud service to a local computer starts with the job itself. Summarising meetings, searching an archive and writing code need different tests. Compare the model, tools and operating conditions of both systems.
Which question are you asking?
Writing code, summarising documents and classifying messages are different tasks with different success criteria. A single average can hide a system that writes useful summaries but fails on code. Keep the results separate before deciding what to move.
Before measuring, write down the decision you need to make. For example, ask whether you can move meeting transcription to a local machine while keeping the quality and turnaround time you need. That gives the trial an outcome to assess.
The minimum protocol
- Representative cases. A set of real examples covering important cases, without a universal minimum count.
- Explicit criteria. What counts as a correct answer, and who decides.
- A held-out set. Keep some cases out of tuning, otherwise you risk adapting the system only to its development examples ( Hugging Face ).
- The whole system. Model version, date, context, enabled tools, hardware for the local case.
- Annotated errors. Not just how many answers are right, but how they fail: a recoverable error and a silent one have different consequences.
What to record, and why
| Dimension | What to measure | Typical mistake |
|---|---|---|
| Quality | cases passed per category | one average across different tasks |
| Latency | time to first token (TTFT) and to completion | measuring only the total |
| Cost | energy, hardware, subscription or API usage, human review time | counting only the monthly price |
| Data | where prompts and outputs end up | assuming “local” is enough |
| Reliability | behaviour on long or ambiguous input | testing only ideal inputs |
Data handling and reliability belong alongside answer quality: they may change the decision even when responses look good.
Read the results by task
A useful conclusion identifies the task and the conditions. For example: “On technical document summaries the local system reaches the quality I need in most cases, within the workflow’s time requirements.” That conclusion needs evidence from the test cases. A claim such as “Local is better” leaves the task and conditions unspecified.
Report failed cases and uncertainties too. If no failures appear, still state the scope, case count and conditions. A limited test does not establish general reliability.
The sample conclusions illustrate how to report a comparison; they are not results from a test we ran. To prepare your own trial, read about the memory budget and data handling in local applications .
When the system takes actions, answer quality is only part of the comparison: assess agent decisions, permissions and errors as well.
Sources
- Evaluation ↗ — Hugging Face · accessed 2026-11-01


