😕 Why S3 Compatible Storage?

In this post on SvelteKit compatible S3 storage, we will take a look at how you can add an upload feature to your Svelte app. We use presigned links, allowing you to share private files in a more controlled way. Rather that focus on a specific cloud storage provider's native API, we take an S3 compatible approach. Cloud storage providers like Backblaze, Supabase and Cloudflare R2 offer access via an API compatible with Amazon's S3 API. The advantage of using an S3 compatible API is flexibility. If you later decide to switch provider, you will be able to keep the bulk of your existing code.

We will build a single page app in SvelteKit which lets the visitor upload a file to your storage bucket. You might use this as a convenient way of uploading files for your projects to the cloud. Alternatively it can provide a a handy starting point for a more interactive app, letting users upload their own content. That might be for a photo sharing app, your own micro-blogging service or for an app letting clients preview and provide feedback on your amazing work. I hope this is something you find interesting if it is let's get going.

⚙️ Getting Started

Let start by creating a new skeleton SvelteKit project. Type the following commands in the terminal:

pnpm init svelte@next sveltekit-graphql-github && cd $_
pnpm install

We will be using the official AWS SDK for some operations on our S3 compatible cloud storage. As well as the npm packages for the SDK we will need a few other packages including some fonts for self-hosting. Lets install all of these now:

pnpm i -D @aws-sdk/client-s3 @aws-sdk/s3-request-presigner @aws-sdk/util-create-request @aws-sdk/util-format-url @fontsource/libre-franklin @fontsource/rajdhani cuid dotenv

Initial Authentication

Although most of the code we look at here should work with any S3 compatible storage provider, the mechanism for initial authentication will be slightly different for each provider. Even taking that into account, it should still make sense to use the provider's S3 compatible API for all other operations to benefit from the flexibility this offers. We focus on Backblaze for initial authentication. Check your own provider's docs for their mechanism.

To get S3 compatible storage parameters from the Backblaze API you need to supply an Account ID and Account Auth token with read and write access to the bucket we want to use. Let's add these to a .env file together with the name of the bucket (if you already have one set up). Buckets offer a mechanism for organising objects (or files) in cloud storage. They play a role analogous to folders or directories on your computer's file system.

S3_COMPATIBLE_ACCOUNT_ID="your-account-id"
S3_COMPATIBLE_ACCOUNT_AUTH_TOKEN="your-auth-token"
S3_COMPATIBLE_BUCKET_NAME="your-bucket-name"

The last bit of setup before spinning up the dev server is to configure the dotenv environment variables package in svelte.config.js:

import 'dotenv/config';

/** @type {import('@sveltejs/kit').Config} */
const config = {
  kit: {
    // hydrate the <div id="svelte"> element in src/app.html
    target: '#svelte',
  },
};

export default config;

Start the dev Server

Use this command to start the dev server:

pnpm run dev

By default it will run on TCP port 3000. If you already have something running there, see how you can change server ports in the article on getting started with SvelteKit.

🔗 Presigned URLs

We will generate presigned read and write URLS on the server side. Presigned URLs offer a way to limit access, granting temporary access. Links are valid for 15  minutes by default. Potential clients, app users and so on will be able to access just the files you want them to access. Also because you are using presigned URLs you can keep the access mode on your bucket set to private.

To upload a file we will use the write signed URL. We will also get a read signed URL. We can use that to download the file if we need to.

Let's create a SvelteKit server endpoint to listen for new presigned URL requests. Create a src/routes/api folder adding an presigned-urls.json.js file with the following content:

import { GetObjectCommand, PutObjectCommand, S3 } from '@aws-sdk/client-s3';
import { S3RequestPresigner } from '@aws-sdk/s3-request-presigner';
import { createRequest } from '@aws-sdk/util-create-request';
import { formatUrl } from '@aws-sdk/util-format-url';
import cuid from 'cuid';

const S3_COMPATIBLE_BUCKET = process.env['S3_COMPATIBLE_BUCKET_NAME'];
const S3_COMPATIBLE_ACCOUNT_AUTH_TOKEN = process.env['S3_COMPATIBLE_ACCOUNT_AUTH_TOKEN'];
const S3_COMPATIBLE_ACCOUNT_ID = process.env['S3_COMPATIBLE_ACCOUNT_ID'];

async function authoriseAccount() {
  try {
    const authorisationToken = Buffer.from(
      `${S3_COMPATIBLE_ACCOUNT_ID}:${S3_COMPATIBLE_ACCOUNT_AUTH_TOKEN}`,
      'utf-8',
    ).toString('base64');

    const response = await fetch('https://api.backblazeb2.com/b2api/v2/b2_authorize_account', {
      method: 'GET',
      headers: {
        Authorization: `Basic ${authorisationToken}`,
      },
    });
    const data = await response.json();
    const {
      absoluteMinimumPartSize,
      authorizationToken,
      apiUrl,
      downloadUrl,
      recommendedPartSize,
      s3ApiUrl,
    } = data;
    return {
      successful: true,
      absoluteMinimumPartSize,
      authorizationToken,
      apiUrl,
      downloadUrl,
      recommendedPartSize,
      s3ApiUrl,
    };
  } catch (error) {
    let message;
    if (error.response) {
      message = `Storage server responded with non 2xx code: ${error.response.data}`;
    } else if (error.request) {
      message = `No storage response received: ${error.request}`;
    } else {
      message = `Error setting up storage response: ${error.message}`;
    }
    return { successful: false, message };
  }
}

This code works for Backblaze's API but will be slightly different if you use another provider. The rest of the code we look at should work with any S3 compatible storage provider.

In lines 7–9 we pull the credentials we stored, earlier, in the .env file. Moving on, in lines 13–16 we see how you can generate a Basic Auth header in JavaScript. Finally, the Backblaze response returns a recommended and minimum part size. These are useful when uploading large files. Typically you will want to split large files into smaller chunks. These numbers give you some guidelines on how big each of the chunks should be. We will look at presigned multipart uploads in another article. Most important though is the s3ApiUrl which we will need to create a JavaScript S3 client.

Creating Presigned Links with S3 SDK

Next we use that S3 API URL to get the S3 region and then use that to get the presigned URLs from the SDK. Add this code to the bottom of the storage.js file:

function getRegion(s3ApiUrl) {
  return s3ApiUrl.split('.')[1];
}

function getS3Client({ s3ApiUrl }) {
  const credentials = {
    accessKeyId: S3_COMPATIBLE_ACCOUNT_ID,
    secretAccessKey: S3_COMPATIBLE_ACCOUNT_AUTH_TOKEN,
    sessionToken: `session-${cuid()}`,
  };

  const S3Client = new S3({
    endpoint: s3ApiUrl,
    region: getRegion(s3ApiUrl),
    credentials,
  });
  return S3Client;
}

async function generatePresignedUrls({ key, s3ApiUrl }) {
  const Bucket = S3_COMPATIBLE_BUCKET;
  const Key = key;
  const client = getS3Client({ s3ApiUrl });

  const signer = new S3RequestPresigner({ ...client.config });
  const readRequest = await createRequest(client, new GetObjectCommand({ Key, Bucket }));
  const readSignedUrl = formatUrl(await signer.presign(readRequest));
  const writeRequest = await createRequest(client, new PutObjectCommand({ Key, Bucket }));
  const writeSignedUrl = formatUrl(await signer.presign(writeRequest));
  return { readSignedUrl, writeSignedUrl };
}

export async function presignedUrls(key) {
  try {
    const { s3ApiUrl } = await authoriseAccount();
    const { readSignedUrl, writeSignedUrl } = await generatePresignedUrls({ key, s3ApiUrl });
    return { readSignedUrl, writeSignedUrl };
  } catch (error) {
    console.error(`Error generating presigned urls: ${error}`);
  }
}

In line 63 we use the cuid package to help us generate a unique session id. That's the server side setup. Next let's look at the client.

🧑🏽 Client Home Page JavaScript

We'll split the code into a couple of stages. First let's add our script block with the code for interfacing with the endpoint that we just created and also the cloud provider. We get presigned URLs from the endpoint then, upload directly to the cloud provider from the client. Since all we need for upload is the presigned URL, there is no need to use a server endpoint. This helps us keep the code simpler.

Replace the content of src/routes/index.svelte with the following:

<script>
  import '@fontsource/rajdhani';
  import '@fontsource/libre-franklin';

  const H_ELLIPSIS_ENTITY = '\\u2026'; // ...
  const LEFT_DOUBLE_QUOTE_ENTITY = '\\u201c'; // "
  const RIGHT_DOUBLE_QUOTE_ENTITY = '\\u201d'; // "

  let isSubmitting = false;
  let uploadComplete = false;
  let files = [];
  let errors = { files: null };
  let downdloadUrl = '';
  $: filename = files.length > 0 ? files[0].name : '';

  function resetForm() {
    files = [];
    errors = { files: null };
  }

  const handleChange = (event) => {
    errors = { files: null, type };
    files = event.target.files;
  };

  const handleSubmit = async () => {
    try {
      if (files.length === 0) {
        errors.files = 'Select a file to upload first';
        return;
      }

      isSubmitting = true;
      const { name: key } = files[0];

      // get signed upload URL
      const response = await fetch('/api/presigned-urls.json', {
        method: 'POST',
        credentials: 'omit',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ key }),
      });
      const json = await response.json();
      const { readSignedUrl, writeSignedUrl } = json;
      downdloadUrl = readSignedUrl;

      // Upload file
      const reader = new FileReader();
      reader.onloadend = async () => {
        await fetch(writeSignedUrl, {
          method: 'PUT',
          body: reader.result,
          headers: {
            'Content-Type': type,
          },
        });
        uploadComplete = true;
        isSubmitting = false;
      };
      reader.readAsArrayBuffer(files[0]);
    } catch (error) {
      console.log(`Error in handleSubmit on / route: ${error}`);
    }
  };
</script>

The first part is mostly about setting up the user interface state. There is nothing unique to this app there, so let's focus on the handleSubmit function. There are two parts. The first in which we get a signed URL from the endpoint we just created and the second where we use the FileReader API to upload the file to the cloud.

FileReader API

The FileReader API lets us read in a file given the local path and output a binary string, DataURL or an array buffer. You would use a DataURL if you wanted to Base64 encode an image (for example). You could then set the src of an <img> element to a generated Base64 data uri string or upload the image to a Cloudflare worker for processing. For our use case, uploading files to cloud storage, instead we go for the readAsArrayBuffer option.

The API is asynchronous so we can just tell it what we want to do once the file is uploaded and carry on living our life in the meantime! We create an instance of the API in line 50. Using onloadend we specify that we want to use fetch to upload our file to the cloud, once it is loaded into an array buffer (from the local file system). In line 62 (after the onreadend block), we specify what we want to read. The file actually comes from a file input, which we will add in a moment.

Fetch Request

The fetch request is inside the onloadend block. We make a PUT request, including the file type in a header. The body of the request is the result of the file read from the FileReader API. Because we are making a PUT request, from the browser, and also because the content type may not be text/plain, we will need some CORS configuration. We'll look at that before we finish.

How do we get the file name and type? When the user selects a file, from the file input we just mentioned, the handleChange code in lines 21–24 runs. This gets the file, by updating the files variable, but does not read the file in (that happens in our FileReader API code). Next, when the user clicks the Upload button which triggers the handleSubmit function call, we get the name and file content type in line 34.

🖥 Client Home Page Markup

Next we'll add the markup, including the file browse input which lets the user select a file to upload. After that we'll add some optional styling, look at CORS rules and finally test.

Paste this code at the bottom of the index.svelte file:

<svelte:head>
  <title>SvelteKit S3 Compatible Storage</title>
  <html lang="en-GB" />
  <meta
    name="description"
    content="Upload a file to third party storage using an S3 compatible API in SvelteKit."
  />
</svelte:head>

<main class="container">
  <h1>SvelteKit S3 Compatible Storage</h1>
  {#if uploadComplete}
    <section class="upload-complete">
      <h2 class="heading">Upload complete</h2>
      <p class="filename">
        Download link: <a aria-label={`Download ${filename}`} href={downdloadUrl}>{filename}</a>
      </p>
      <div class="button-container">
        <button
          class="another-upload-button"
          on:click={() => {
            uploadComplete = false;
            resetForm();
          }}>Upload another file</button
        >
      </div>
    </section>
  {:else}
    <section class="upload">
      <form on:submit|preventDefault={handleSubmit}>
        <h2 class="heading">Upload a file{H_ELLIPSIS_ENTITY}</h2>
        {#if filename !== ''}
          <p class="filename">{filename}</p>
          <p class="filename">
            Click {LEFT_DOUBLE_QUOTE_ENTITY}Upload{RIGHT_DOUBLE_QUOTE_ENTITY} to start upload.
          </p>
        {/if}
        {#if errors.files}
          <div class="error-text-container">
            <small id="files-error" class="error-text">{errors.files}</small>
          </div>
        {/if}
        {#if isSubmitting}
          <small id="files-error">Uploading{H_ELLIPSIS_ENTITY}</small>
        {/if}
        <div class="file-input-container">
          <label class="file-input-label" for="file"
            ><span class="screen-reader-text">Find a file to upload</span></label
          >
          <input
            id="file"
            aria-invalid={errors.files != null}
            aria-describedby={errors.files != null ? 'files-error' : null}
            type="file"
            multiple
            formenctype="multipart/form-data"
            accept="image/*"
            title="File"
            on:change={handleChange}
          />
          <div class="button-container">
            <button type="submit" disabled={isSubmitting}>Upload</button>
          </div>
        </div>
      </form>
    </section>
  {/if}
</main>

You can see the file input code in lines 118–128. We have set the input to allow the user to select multiple files (multiple attribute in line 123). For simplicity the logic we added previously only uploads the first file, though you can tweak it if you need multiple uploads from your application. In line 125 we set the input to accept only image files with accept="image/*". This can be helpful for user experience, as typically in the file select user interface, just image files will be highlighted. You can change this to accept just a certain image format or different file types, like PDF, or video formats — whatever your application needs. See more on file type specifier in the MDN docs.

Finally before we check out CORS, here's some optional styling. This can be nice to add as the default HTML file input does not look a little brutalistic!

<style>
  :global(html) {
    background-image: linear-gradient(
      to top right,
      var(--colour-theme-lighten-20),
      var(--colour-theme)
    );
    color: var(--colour-light);

    font-family: Libre Franklin;
  }

  :global(:root) {
    --colour-theme: #3185fc; /* azure */
    --colour-theme-lighten-20: #4599ff;
    --colour-light: #fafaff; /* ghost white */
    --colour-light-opacity-85: #fafaffd9;
    --colour-dark: #403f4c; /* dark liver */
    --colour-feature: #f9dc5c; /* naples yellow */
    --colour-alternative: #e84855; /* red crayola */
    --font-weight-medium: 500;
  }

  .screen-reader-text {
    border: 0;
    clip: rect(1px, 1px, 1px, 1px);
    clip-path: inset(50%);
    height: 1px;
    margin: -1px;
    width: 1px;
    overflow: hidden;
    position: absolute !important;
    word-wrap: normal !important;
  }
  .error-text-container {
    margin: 2rem 0 0.5rem;
  }
  .error-text {
    color: var(--colour-feature);
    background-color: var(--colour-dark);
    padding: 0.5rem 1.25rem;
    border-radius: 1.5rem;
    border: solid 0.0625rem var(--colour-feature);
  }

  .container {
    margin: 1.5rem;
    min-height: 100vh;
  }

  .container h1 {
    font-family: Rajdhani;
    font-size: 1.953rem;
  }

  .upload,
  .upload-complete {
    margin: 4rem 1rem;
    padding: 1.5rem;
    border: solid 0.125rem var(--colour-light);
    border-radius: 0.5rem;
  }

  .button-container {
    display: flex;
  }

  :is(.upload, .upload-complete) .heading {
    font-family: Rajdhani;
    font-size: 1.563rem;
    margin-top: 0;
  }

  .upload-complete {
    background-color: var(--colour-feature);
    color: var(--colour-dark);
    border-color: var(--colour-dark);
  }
  .filename {
    margin-left: 1rem;
  }

  .filename a {
    color: var(--colour-dark);
    text-underline-offset: 0.125rem;
  }

  .file-input-container {
    display: flex;
    align-items: center;
    justify-content: flex-end;
    padding: 1.5rem 0 0.5rem;
  }

  .file-input-label::before {
    content: 'Browse\\2026';
    margin-left: auto;
  }

  .file-input-label::before,
  button {
    font-family: Libre Franklin;
    background: var(--colour-theme);
    cursor: pointer;
    color: var(--colour-light);
    border: solid 0.0625rem var(--colour-light);
    border-radius: 1.5rem;
    margin-left: 1rem;
    padding: 0.5rem 1.75rem;
    font-size: 1.25rem;
    font-weight: var(--font-weight-medium);
  }

  @media (prefers-reduced-motion: no-preference) {
    .file-input-label::before,
    button {
      transition: background-color 250ms, color 250ms;
    }
  }
  @media (prefers-reduced-motion: no-preference) {
    .file-input-label::before,
    button {
      transition: background-color 2000ms, color 2000ms;
    }
  }

  button:hover,
  .file-input-label:hover:before,
  button:focus,
  .file-input-label:focus:before {
    background-color: var(--colour-light-opacity-85);
    color: var(--colour-dark);
  }

  .another-upload-button {
    margin-left: auto;
  }

  .upload-complete button:hover,
  .upload-complete button:focus {
    border-color: var(--colour-dark);
  }

  input[type='file'] {
    visibility: hidden;
    width: 1px;
  }

  @media (min-width: 768px) {
    .container {
      margin: 3rem 1.5rem;
    }

    .upload,
    .upload-complete {
      margin: 4rem 10rem;
    }
  }
</style>

⛔️ Cross-Origin Resource Sharing (CORS)

CORS rules are a browser security feature which limit what can be sent to a different origin. By origin we mean sending data to example-b.com when you are on the example-a.com site. If the request to a cross origin does not meet some basic criteria (GET request or POST with text/plain content type, for example) the browser will perform some extra checks. We send a PUT request from our code so the browser will send a so-called preflight request ahead of the actual request. This just checks with the site we are sending the data to what it is expecting us to send, or rather what it will accept.

To avoid CORS issues, we can set CORS rules with our storage provider. It is possible to set them on your bucket when you create it. Check with your provider on the mechanism for this. With Backblaze you can set CORS rules using the b2 command line utility in JSON format. Here is an example file:

[
  {
    "corsRuleName": "development",
    "allowedOrigins": ["https://test.localhost.com:3000"],
    "allowedHeaders": ["content-type", "range"],
    "allowedOperations": ["s3_put"],
    "exposeHeaders": ["x-amz-version-id"],
    "maxAgeSeconds": 300
  },
  {
    "corsRuleName": "production",
    "allowedOrigins": ["https://example.com"],
    "allowedHeaders": ["content-type", "range"],
    "allowedOperations": ["s3_put"],
    "exposeHeaders": ["x-amz-version-id"],
    "maxAgeSeconds": 3600
  }
]

We can set separate rules to let our dev and production requests work. In the allowed origin for dev, we set a dummy hostname instead of localhost and on top we run in HTTPS mode. You may be able to have everything working without this setup, but try it if you have issues. Add this CORS configuration to Backblaze with the CLI utility installed by running:

b2 update-bucket --corsRules "$(cat backblaze-bucket-cors-rules.json)" your-bucket-name allPrivate

You can see more on Backblaze CORS rules in their documentation.

Secure dev Server

To run the SvelteKit dev server in https mode, update your package.json dev script to include the --https flag:

{
  "name": "sveltekit-s3-compatible-storage",
  "version": "0.0.1",
  "scripts": {
    "dev": "svelte-kit dev --port 3000 --https",

Then restart the dev server with the usual pnpm run dev command. Learn more about this in the video on running a secure SvelteKit dev server.

To set a local hostname, on MacOS add a line to private/etc/hosts:

  127.0.0.1 test.localhost.com

Then, instead of accessing the site via http://localhost:3030, in your browser use https://test.localhost.com:3030. This worked for me on macOS. The same will work on typical Linux and Unix systems, though the file you change will be /etc/hosts. If you are using DNSCryprt Proxy or Unbound, you can make a similar change in the relevant config files. If you use windows and know how to do this, please drop a comment below to help out other windows users.

💯 SvelteKit S3 Compatible Storage: Test

Try uploading a file using the new app. Also make sure the download link works.

🙌🏽 SvelteKit S3 Compatible Storage: What we Learned

In this post we learned:

• why you would use the S3 compatible API for cloud storage instead of your storage provider's native API,

• how to use the AWS SDK to generate a presigned upload URL,

• a way to structure a file upload feature in a SvelteKit app.

I do hope there is at least one thing in this article which you can use in your work or a side project. As an extension you might want to pull a bucket list and display all files in the folder. You could even add options to delete files. On top, you could also calculate a hash of the file before upload and compare that to the hash generated by your storage provider. This avails a method to verify file integrity. There's a world of different apps you can add an upload feature to; knock yourself out!

You can see the full code for this SvelteKit S3 compatible storage project on the Rodney Lab Git Hub repo.

🙏🏽 SvelteKit S3 Compatible Storage: Feedback

Have you found the post useful? Would you prefer to see posts on another topic instead? Get in touch with ideas for new posts. Also if you like my writing style, get in touch if I can write some posts for your company site on a consultancy basis. Read on to find ways to get in touch, further below. If you want to support posts similar to this one and can spare a few dollars, euros or pounds, please consider supporting me through Buy me a Coffee.

Finally, feel free to share the post on your social media accounts for all your followers who will find it useful. As well as leaving a comment below, you can get in touch via @askRodney on Twitter and also askRodney on Telegram. Also, see further ways to get in touch with Rodney Lab. I post regularly on SvelteKit as well as other topics. Also subscribe to the newsletter to keep up-to-date with our latest projects.