Skip to main content

1. Background​

In an instant messaging system, images, voice messages, videos, files, video covers, and similar content are all object data. They are different from normal text messages: they are usually larger, take longer to upload, and are more sensitive to network fluctuations. If every file is relayed through the business API service, the server has to handle both business processing and high-volume file transfer, which can easily affect the stability of core paths such as messages, conversations, and groups.

OpenIM's S3 object storage capability is designed to solve this problem: business services are responsible only for authentication, signatures, metadata, and access control, while file data goes directly into object storage. This improves upload efficiency and reduces pressure on API services.

2. Core Positioning​

OpenIM's S3 solution is not simply about uploading files to MinIO. It is a large-object management capability built for IM scenarios. It covers upload, download, access, resumable upload, deduplication, cleanup, migration, and related stages.

At a high level, it can be understood as two layers:

  • The server is responsible for the control plane: authentication, temporary credential generation, object information registration, unified access address generation, and expired-object cleanup.
  • The SDK is responsible for the client upload experience: reading files, calculating hashes, multipart upload, resumable upload, concurrency control, progress callbacks, and upload cancellation.

This split separates "business control" from "file traffic". The server no longer moves large files; after receiving authorization, the client uploads directly to object storage.

3. Overall Upload Experience​

When a user sends an image, video, or file, the SDK first analyzes the file locally, determines its size and type, and calculates content features for the file. The SDK then requests upload authorization from the server. After the server verifies the user's identity and the object's ownership, it returns temporary upload credentials. The file content is then uploaded directly from the client to object storage, without being relayed through the OpenIM API service.

After the upload is complete, the server registers the relationship between this object and the business object name, and returns an OpenIM-level access address. Business messages store this stable address, not the raw address exposed by a storage vendor. During later access, OpenIM generates a temporary access link based on object metadata and redirects to the real object.

This flow has several benefits:

  • Uploading large files does not consume business API bandwidth.
  • Download addresses can be managed centrally by OpenIM.
  • Business-layer addresses can remain stable when the underlying storage backend changes.
  • Private buckets can also be accessed securely through temporary authorization.

4. Compatibility With Multiple Storage Backends​

OpenIM supports multiple S3 or S3-compatible object storage services, including MinIO, Tencent Cloud COS, Alibaba Cloud OSS, Qiniu Kodo, AWS S3, and others. The business layer does not need to understand vendor-specific differences; it only needs to use the unified object capability provided by OpenIM.

This makes deployment choices more flexible:

  • Private deployments can use MinIO.
  • Public cloud deployments can connect to cloud-vendor object storage.
  • If an enterprise later switches storage vendors, the business logic does not need to be rewritten as a whole.
  • Storage access domains, internal and external network addresses, and permission policies can be adjusted for each deployment environment.

5. Direct Client Upload​

A common approach for traditional file upload is for the client to upload the file to the business service first, and then for the business service to transfer it to object storage. This is simple to implement, but it is costly for large files and high concurrency.

OpenIM uses direct client upload: the server only issues short-lived authorization, and the file itself is uploaded directly from the client to object storage. As a result, API services only process lightweight requests and do not handle large file transfer.

Direct client upload has these advantages:

  • Reduces API service bandwidth and connection pressure.
  • Makes large file upload more stable without affecting core interfaces such as message sending and receiving.
  • Object storage is naturally suited for large objects and high-concurrency downloads.
  • The file upload path is shorter, improving overall throughput.

6. Multipart Upload​

For larger images, videos, and files, the SDK uses multipart upload. The file is split into multiple smaller parts, each part is uploaded independently, and object storage merges them into a complete object at the end.

Multipart upload brings a clear improvement in user experience:

  • Large files do not need to be uploaded all at once.
  • If one part fails, only the failed part needs to be retransmitted.
  • Upload speed can be improved through controlled concurrency.
  • The upload process can show more fine-grained progress.

The SDK automatically selects an appropriate part size based on file size and server-side limits, so upper-layer business code does not need to handle these details manually.

7. Instant Upload​

OpenIM supports instant upload based on file content. The SDK calculates content features for the file, and the server can determine whether the same content already exists. If object storage already has this file, the file content is not uploaded again. Instead, the server directly registers a new business reference and returns an access address.

Instant upload is especially valuable in repeated sending scenarios:

  • When the same user sends the same file again, the upload can complete instantly.
  • Different messages can reference the same real object, reducing duplicate storage.
  • Network transfer cost and object storage write cost are reduced.
  • Unnecessary retransmission is reduced in weak network environments.

Instant upload is based on content features, not file names, which makes it better suited to IM file scenarios.

8. Resumable Upload​

The SDK records upload progress locally. During upload, if the network is interrupted, the app exits, or the process restarts, successfully uploaded parts are not lost. When the user uploads the same file again, the SDK can restore the local state and upload only the remaining unfinished parts.

Resumable upload significantly improves weak-network and mobile experiences:

  • Large files do not need to restart from the beginning after a failure.
  • Uploads can continue after backgrounding, network interruption, or restart.
  • Every successfully uploaded part becomes valid progress.
  • Retrying is cheaper for users, and failure behavior is more controllable.

This is also one of the important advantages of OpenIM's S3 solution compared with normal form upload.

9. Concurrency and Memory Control​

The SDK supports concurrent multipart upload, but it does not allow unbounded concurrency. During upload, it dynamically adjusts the number of concurrent uploads based on part size and memory budget, avoiding excessive client memory usage for the sake of speed.

When parts are smaller, the SDK can upload multiple parts concurrently. When parts are larger, the SDK lowers concurrency. If parts are too large, it falls back to a more conservative streaming upload method.

This strategy is more stable across mobile, desktop, Web/WASM, and other environments. It balances upload speed, memory usage, and device stability.

10. Progress Feedback and Upload Cancellation​

The SDK breaks the upload process into multiple perceivable stages, such as opening the file, calculating hashes, obtaining an upload session, uploading parts, and completing the merge. Upper-layer UI can use this to display more accurate progress instead of a vague percentage.

This is important for large files. Users can clearly understand whether the current step is local calculation, network upload, or waiting for the server to complete the merge.

The SDK also supports upload cancellation. The business side can bind a cancellation identifier to an upload task. After the user clicks cancel, the SDK stops the corresponding upload task to avoid continuing to consume network and device resources.

11. Unified Access Address​

After an upload is complete, OpenIM returns a unified OpenIM access address instead of directly exposing the raw object storage address. When accessing a file, OpenIM generates a temporary access link based on object metadata and then redirects to the real storage address.

This design has several benefits:

  • File addresses stored in messages are more stable.
  • Underlying storage migration does not directly affect historical messages.
  • Private objects can also be accessed securely through temporary authorization.
  • Access policies such as file names, content types, and image thumbnails can be handled centrally.

For image objects, the access layer can also support derived capabilities such as thumbnails, format conversion, and width or height cropping, improving image loading efficiency in chats.

12. Permissions and Isolation​

When a normal user uploads an object, the object is placed in that user's own namespace. This prevents users from forging or overwriting other users' file objects. Before generating upload authorization, the server verifies identity and object ownership to ensure the upload behavior is controlled.

Long-term object storage keys are not issued to clients. The client receives only short-lived upload or access credentials. Even if a link is leaked, the risk is limited to a short time window and a specific object scope.

This approach balances direct upload efficiency with security boundaries.

13. Metadata Governance​

Object storage is responsible only for storing file content, but an IM system also needs to know which user the file belongs to, which business type it belongs to, which storage backend it uses, when it was created, whether it can be cleaned up, and other information.

OpenIM registers metadata for each business object. Metadata separates the "business object name" from the "real storage object". Multiple business objects can reference the same real object, which is also the foundation for instant upload and deduplication.

Metadata governance provides the following capabilities:

  • Generate access addresses uniformly.
  • Support content deduplication and instant upload.
  • Support lifecycle cleanup by business type.
  • Support checking whether a real object is still referenced.
  • Support future migration of the underlying storage.

14. Lifecycle Cleanup​

Not every file object in an IM system needs to be stored forever. For example, message objects such as images, voice messages, videos, and video covers can have retention periods based on business policies.

OpenIM supports cleaning up expired objects by business category. During cleanup, it does not simply delete the real file. Instead, it deletes expired business references first, and then checks whether the real object is still referenced by other business objects. Only when there are no references left is the real file in object storage deleted.

This reference-counting cleanup avoids accidental deletion. It naturally works together with instant upload: the same real file may be referenced by multiple messages, and the real object should be deleted only after all references have expired.

15. FormData Compatibility Channel for Mini Programs​

In addition to the complete SDK multipart upload path, OpenIM also keeps a FormData direct form upload capability. This path is mainly used by the mini program JSSDK to adapt to file upload capabilities and invocation limits on mini program platforms.

It is important to note that the SDK provided by Go does not call FormData. The Go SDK uses the complete multipart upload path.

FormData is a compatibility channel and does not provide the advanced capabilities of the complete upload path. It does not support:

  • Multipart upload.
  • Instant upload.
  • Local resumable upload.
  • Multipart concurrency.
  • Part-level verification or part-level progress callbacks.

Therefore, normal apps, desktop clients, the Go SDK, and Web/WASM clients should prioritize the complete SDK upload capability. Under platform limits, the mini program JSSDK can use FormData as a simplified direct upload solution.

16. Storage Migration Capability​

Because OpenIM records the storage backend that each object belongs to in metadata, the system has the foundation for migrating from one object storage service to another. For example, it can migrate from a self-hosted MinIO deployment to cloud-vendor object storage, or switch between different cloud vendors.

During migration, the system can read the real object from the old storage, write it to the new storage, and update the storage backend information in metadata. Business messages store OpenIM's unified access address, so historical messages can still be accessed in the original way after migration.

This reduces long-term operations risk and lowers the cost of being tied to a single storage vendor or deployment model.

17. Advantages Over Traditional Upload Methods​

The advantages of OpenIM's S3 solution can be summarized in five categories.

In terms of performance, file traffic goes directly to object storage, and API services only process control requests. Large file uploads do not consume core business interfaces.

In terms of stability, multipart upload, resumable upload, part verification, upload cancellation, and controlled concurrency work together to improve the success rate in weak network environments.

In terms of cost, instant upload and deduplication reduce duplicate uploads and duplicate storage, while lifecycle cleanup reduces long-term space occupied by invalid objects.

In terms of security, clients receive only short-lived authorization and never touch long-term object storage keys. File access can also be controlled through temporary links.

In terms of evolution, unified object addresses and metadata governance allow the underlying storage to be migrated or replaced without requiring the business layer to understand vendor-specific changes.

18. Applicable Scenarios​

OpenIM S3 object storage is suitable for the following scenarios:

  • Chat images, original images, and thumbnails.
  • Voice messages.
  • Video messages and video covers.
  • Normal file messages.
  • Capabilities that require temporary object upload, such as ASR.
  • File storage in private deployments.
  • Large file, weak network, and mobile upload scenarios.

If the business only needs simple form upload for mini programs, it can use the FormData compatibility channel. If instant upload, multipart upload, resumable upload, and a more complete upload experience are required, it should use the complete SDK upload capability.

19. Summary​

OpenIM's S3 object storage capability is file infrastructure built for large-object IM scenarios. It reduces server pressure through direct client upload, improves weak-network experience through multipart and resumable upload, lowers costs through instant upload and lifecycle cleanup, and improves long-term maintainability through unified access addresses and metadata governance.

Its core value is to turn file upload from "a heavy burden on the business API" into "an object service capability that can be authorized, resumed, deduplicated, governed, and migrated". For enterprise-grade IM, this design improves performance, stability, security, cost control, and future evolution at the same time.