Improving Durability of the Blog

I've mentioned previously that my database is SQLite, an embedded database.

That means the database runs on the same machine as my code. It also means that if my code were to lose access to the physical storage of the machine it's running on somehow, we could lose our DB.

To account for this, I deployed my app using a Digital Ocean VPS. A VPS has some capabilities that make it more suitable for persistent storage like this. In case of reboots or a physical drive failure, data will be safe.

However, there are still scenarios where my SQLite DB could be wiped:

• Destruction of the instance
• Rebuild of the instance
• (less likely) a data center fire

Because of this, I decided to implement a simple backup mechanism for the blog's db.

The Backup Method

Our method and implementation is simple - periodically, the app will write the DB to cloud storage, in this case S3. This makes our backups automated. In the case that my SQLite DB is lost, I can simply copy the backup from S3 back to my VPS.

Implementation

To implement this, we need a way to interface with object storage like S3. Thankfully, there are libraries that do this for us.

First, we create a client that knows how to interact with cloud storage like S3.

 1func NewSpaceClient() (*SpaceClient, error) {
 2	if os.Getenv("ENV") != "prod" {
 3		return nil, fmt.Errorf("environment is not prod - not configuring backups")
 4	}
 5	key := os.Getenv("SPACES_KEY")
 6	secret := os.Getenv("SPACES_SECRET")
 7	endpoint := os.Getenv("SPACES_ENDPOINT")
 8	region := os.Getenv("SPACES_REGION")
 9	bucket := os.Getenv("SPACES_BUCKET")
10
11	creds := credentials.NewStaticCredentialsProvider(key, secret, "")
12
13	cfg, err := config.LoadDefaultConfig(context.TODO(),
14		config.WithRegion(region),
15		config.WithCredentialsProvider(creds),
16	)
17	if err != nil {
18		return nil, fmt.Errorf("unable to load SDK config, %v", err)
19	}
20
21	client := s3.NewFromConfig(cfg, func(o *s3.Options) {
22		o.BaseEndpoint = aws.String(endpoint)
23	})
24
25	return &SpaceClient{
26		Client: client,
27		Bucket: bucket,
28	}, nil
29}

This struct has a method that knows how to upload a file to the cloud storage:

 1func (s *SpaceClient) UploadFile(ctx context.Context, objectKey string, fileReader io.Reader) error {
 2	_, err := s.Client.PutObject(ctx, &s3.PutObjectInput{
 3		Bucket: aws.String(s.Bucket),
 4		Key:    aws.String(objectKey),
 5		Body:   fileReader,
 6		ACL:    "private",
 7	})
 8	if err != nil {
 9		return fmt.Errorf("failed to upload file: %w", err)
10	}
11	return nil
12}

Now, we need a worker that will periodically call this UploadFile method. A goroutine works perfectly for this.

 1type BackupService struct {
 2	spaceClient *backup.SpaceClient
 3	dbPath      string
 4	interval    time.Duration
 5}
 6
 7func NewBackupService(client *backup.SpaceClient, dbPath string, interval time.Duration) *BackupService {
 8	return &BackupService{
 9		spaceClient: client,
10		dbPath:      dbPath,
11		interval:    interval,
12	}
13}
14
15func (b *BackupService) Start(ctx context.Context) {
16	ticker := time.NewTicker(b.interval)
17
18	go func() {
19		for {
20			select {
21			case <-ctx.Done():
22				ticker.Stop()
23				return
24			case <-ticker.C:
25				if err := b.performBackup(ctx); err != nil {
26					log.Printf("Backup failed: %v", err)
27				} else {
28					log.Printf("Backup successful")
29				}
30			}
31		}
32	}()
33}
34
35func (b *BackupService) performBackup(ctx context.Context) error {
36	f, err := os.Open(b.dbPath)
37	if err != nil {
38		return err
39	}
40	defer f.Close()
41
42	filename := "backups/blog.db"
43
44	return b.spaceClient.UploadFile(ctx, filename, f)
45}

Given the interval, in the Start method, we create a new goroutine that wakes up every interval seconds and uploads the backup, via this block:

1case <-ticker.C:
2	if err := b.performBackup(ctx); err != nil {
3		log.Printf("Backup failed: %v", err)
4	} else {
5		log.Printf("Backup successful")
6	}
7}

Now, we need to wire the whole thing up into our main.go file, so that it starts the worker when the server is started.

 1	package main
 2
 3	backupCtx, cancelBackup := context.WithCancel(context.Background())
 4	defer cancelBackup()
 5
 6	backupClient, err := backup.NewSpaceClient()
 7	if err != nil {
 8		log.Printf("error getting S3 client: %v", err)
 9	} else {
10		backupWorker := tasks.NewBackupService(backupClient, "blog.db", time.Hour)
11		backupWorker.Start(backupCtx)
12	}

This code executes just after we start the actual server.

Conclusion

Now our blog has automated backups, and the VPS is no longer a single point of failure.