How to Prevent Duplicate Records in Celigo Integrations

Make Every Repeat Recognize the Same Record

A duplicate order can trigger a second shipment. A duplicate customer can split purchase history across two profiles. To prevent these problems in Celigo, give each business record a stable key, use it to find existing records, and make repeated writes safe.

We conducted a documentation review of Celigo, Oracle NetSuite, AWS, and Stripe guidance for this article. Our analysis focuses on record matching, retries, and concurrent writes. The examples below are design suggestions and a proposed test plan, not results from a live customer study.

1. Trace Where the Second Record Starts

Choose one confirmed duplicate pair. Compare the source ID, destination IDs, creation times, flow runs, and retry history. Check whether two different flows can create the same record. Include manual imports and backfills in that review.

Repeated delivery is a real integration case. Stripe says webhook endpoints can receive an event more than once and recommends tracking processed event IDs.^[6] That is a source-specific example of why a flow should handle repeated input.

• Same source ID, two destination records: inspect the matching rule and overlapping writes.
• Different source IDs, same person: review source data and customer matching rules.
• Duplicates after a timeout: check whether the first write succeeded before the retry.
• Duplicates during a backfill: check for overlap with the live flow.

Keep a short incident note with the affected keys and the flow version. This gives the team a concrete case to reproduce after changing the configuration.

2. Choose a Key That Survives a Retry

Use an ID that stays the same when the record is edited or resent. For example, an order key might be store-a:order:84721. A second store needs a different prefix even if it also has order 84721. Do not generate a new timestamp or random ID on each attempt.

Oracle recommends external IDs with SOAP upsert operations to help prevent duplicates. Its documentation also notes that some record types do not support external IDs and that uniqueness can span record groups.^[2] Check your record type and API before applying that pattern.

Reject blank keys before the write. Keep formatting consistent, including spaces and leading zeros. Decide who owns the NetSuite external ID before changing it. Oracle advises a single approach to maintaining those values.^[2] If another integration owns that field, agree on a separate mapping rather than overwriting it.

3. Match the Import Action to the Business Rule

Celigo documents Add, Update, and Add or update for NetSuite imports. For an Add operation, Ignore existing records can skip records found by the matching criteria. Its How can we find existing records? settings apply to Celigo RESTlets imports; REST API imports configure lookup logic differently.^[1]

Choose whether a repeat should update or skip the existing record. Updating a customer address may be useful. Replacing fields on an already approved transaction may be wrong. Write that decision down before selecting an operation.

For supported HTTP imports, Celigo's Composite method offers create-and-ignore, create-and-update, and update-only choices. A dynamic lookup can identify an existing destination record.^[3] Confirm which options your connector exposes; these settings are not identical across every application.

Test the lookup with three inputs: no match, one match, and more than one match. Our recommended rule is to hold ambiguous matches for review. A lookup failure should also stop the write, rather than be treated as proof that no record exists.

A Five-Step Duplicate Prevention Flow

Use this sequence as a design checklist. The matching step controls whether the record reaches a write at all.

4. Protect Records Written at the Same Time

Consider two requests for the same order. Both search before either creates a record. Both may see no match. This is why a lookup alone is not a complete concurrency safeguard.

Celigo documents a Concurrency ID lock template under an import's advanced settings. Records with the same evaluated key are processed in sequence, while unrelated keys can still run in parallel.^[4] Where supported, base that lock on the same business key used for matching and inspect the evaluated value with sample data.

Do not assume an import lock coordinates every other flow, CSV upload, or application. Review all writers. Prefer a destination-enforced unique key or an atomic write operation where supported. If the destination rejects a duplicate key, find and verify the existing record before treating the attempt as recovered.

5. Make Retries Safe After an Unclear Result

A timeout can leave the outcome unknown: the destination may have saved the record even though the response did not arrive. AWS describes using caller-provided request IDs so a service can recognize repeated attempts at the same operation.^[5] This is the purpose of idempotency: repeating an operation should not repeat its business effect.

For an HTTP API that supports idempotency keys, reuse the original request key for a retry of that same action. Follow the API's rules for key lifetime and changed payloads. Adding an arbitrary header to an API that does not support it provides no protection.

For record imports, keep the original matching key and check the destination after an unclear response. For a multi-step flow, verify each completed step before resuming. Finding one existing order does not prove that its payment or fulfillment step succeeded.

• Keep the source key, destination ID, and operation outcome together.
• Separate a repeated event from a new update to the same record.
• Use a distinct action key for a new refund or payment.
• Check side effects such as emails and shipments during replay tests.

Stripe distinguishes repeated event delivery from separate events about the same object.^[6] Track event identity for delivery handling and record identity for destination matching. Do not permanently suppress every future change just because that customer or order was processed once.

6. Prove the Rules With a Small Replay Test

Run a controlled test in a sandbox before changing the live flow. Start with an order that has a known source key. Send it once, then resend the same input. Check both the destination record count and the business result.

Save the inputs, settings, run IDs, and resulting destination IDs. These are proposed acceptance checks, not reported test results. Repeat the relevant cases after changes to keys, lookup rules, or retry handling.

7. Watch for Duplicates After Launch

Schedule a destination report grouped by the chosen source key. Review keys linked to more than one record, missing keys, and unresolved matching errors. Set the review frequency to fit the cost of a duplicate in your process.

Preventing new duplicates does not repair old ones. Have the business owner choose the correct record and review linked transactions before merging, voiding, or removing anything. Then repair the source-to-destination mapping so the next run finds the right record.

For a wider review, use our guide to auditing an existing Celigo integration. For retry ownership and daily support, see how to reduce Celigo integration errors.

References

Official product documentation and engineering guidance reviewed September 30, 2026.

1. Celigo: Import data into NetSuite.
2. Oracle NetSuite: External IDs overview.
3. Celigo: Identify existing records when importing records.
4. Celigo: Preserve the order of records when using concurrency.
5. AWS Builders’ Library: Making retries safe with idempotent APIs.
6. Stripe: Handle duplicate webhook events.