Safely Migrating Historical Encrypted Outlook Email Threads into SDP requests
Note: This post was generated with AI assistance based on a lengthy, real-world Outlook → ServiceDesk Plus migration script. The goal is to share the underlying design and lessons learned in a concise, reusable way.
We recently needed to migrate a large backlog of historical Outlook email support threads into ManageEngine ServiceDesk Plus (on-prem). The migration had several constraints:
-
Emails were spread across multiple folders (custom folders, Inbox, Sent, Deleted)
-
Emails lived in multiple mailboxes
-
Encryption, forwards, and replies caused Outlook ConversationID values to diverge
The migration had to be safe to rerun multiple times without creating duplicate SDP requests or duplicate notes.
SDP workflows and templates populate notify fields after ticket creation
Using Outlook ConversationID or SDP’s email connector did not work reliably for this scenario.
This approach is focused on idempotent backlog migration, not real-time email ingestion.
-
Why ConversationID was unreliable
Outlook ConversationID is not stable when:
-
messages are encrypted
-
messages are forwarded instead of replied
-
threads cross mailbox boundaries
In testing, a single human-visible conversation often produced multiple ConversationIDs, which led to duplicate tickets.
-
Strategy A: RFC header–based thread grouping
Instead of ConversationID, threads are grouped using RFC transport headers available via Outlook MAPI:
-
PR_TRANSPORT_MESSAGE_HEADERS (proptag 0x007D001E)
The thread key is derived in this order:
-
References: → first Message-ID in the chain (preferred root)
-
In-Reply-To:
-
Message-ID:
-
Fallback only if headers are unavailable
This produces a stable key such as:
A_REFROOT::<message-id>
This approach survived encryption, forwards, folder moves, and cross-mailbox scenarios.
-
Mailbox-qualified thread mapping (prevents duplicate tickets)
To safely rerun migrations and handle multiple mailboxes, the thread key is qualified with the mailbox name:
MAILBOX::<mailbox_display_name>::THREAD::<strategy_a_key>
A persistent JSON map stores:
(mailbox + thread key) → SDP request ID
On reruns:
-
if the key already exists, the script reuses the existing request
-
no duplicate SDP requests are created
-
Preventing duplicate notes across reruns
A second idempotency layer prevents duplicate notes.
For each SDP request, a small checkpoint file records which Outlook message EntryIDs have already been added as notes.
On rerun:
-
messages already in the checkpoint are skipped
-
new messages in the same thread are added as notes
This allows multiple runs, partial runs, and incremental additions without duplication.
-
Performance optimization: folder-scoped discovery
Instead of scanning entire mailboxes:
-
specific “seed” folders define which threads matter
-
Inbox, Sent, and Deleted are scanned only to pull additional messages for those threads
This dramatically reduced runtime compared to full mailbox scans.
-
Workflow-safe notify updates
Because SDP workflows/templates may populate notify fields after ticket creation:
-
the script waits briefly before updating notify
-
merges workflow-added values with expected notify recipients
-
reapplies safely without overwriting workflow behavior
What I’m sharing with the community:
-
A sanitized README explaining the design and configuration
-
Targeted code excerpts illustrating the key techniques
I’m intentionally not posting the full production script inline, but happy to share patterns or answer questions if this is useful.
Excerpt 1: Strategy A – RFC header–based thread key
Purpose: show why this is more reliable than Outlook ConversationID.
CODE:
- headers = get_transport_headers(mail_item)
- if headers.references:
- # Use first Message-ID as stable root
- thread_key = "A_REFROOT::" + first_message_id(headers.references)
- elif headers.in_reply_to:
- thread_key = "A_INREPLY::" + headers.in_reply_to
- elif headers.message_id:
- thread_key = "A_MSGID::" + headers.message_id
- else:
- thread_key = "A_FALLBACK::" + mail_item.EntryID
Why this matters:
-
Survives encryption, forwards, and mailbox boundaries
-
Deterministic across folders and reruns
Excerpt 2: Mailbox-qualified thread map (ticket deduplication)
Purpose: show how duplicate SDP requests are prevented across mailboxes and runs.
CODE:
- map_key = (
- "MAILBOX::" + mailbox_display_name +
- "::THREAD::" + strategy_a_thread_key
- )
- if map_key in conversation_map:
- request_id = conversation_map[map_key]
- else:
- request_id = create_sdp_request(...)
- conversation_map[map_key] = request_id
- save_conversation_map(conversation_map)
Why this matters:
-
Same thread key in different mailboxes cannot collide
-
Safe to run repeatedly and incrementally
Excerpt 3: “Already-noted” checkpoint (note deduplication)
Purpose: prevent duplicate notes on reruns.
CODE:
- already_noted = load_already_noted(request_id)
- for message in followup_messages:
- if message.EntryID in already_noted:
- continue
- add_note_to_request(request_id, message)
- already_noted.add(message.EntryID)
- save_already_noted(request_id, already_noted)
Why this matters:
-
Reruns do not duplicate notes
-
Partial runs can resume safely
-
New emails are appended cleanly
I’m sharing this primarily as a design pattern; feedback or alternative approaches are welcome.