Tracking down large metadata changes

rawtaz · April 25, 2021, 9:28pm

I have a macOS 11.2.3 (but the problem persisted even with 10.4.6) client where every backup that’s taken every hour adds around 400-450 MB new data to the repository. I’m trying to nail down what it is that’s generating this much changed data, because the stuff the user herself changes shouldn’t be more than a few tens of MBs.

I am able to reproduce the problem fully, by running a backup, and even if I do so with just a couple of minutes between the runs, the problem appears, although if I do it that quickly the amount of changes is more around 380 MB instead of the 400-450 MB over an hour. This goes hand in hand with how much data changes on other client’s I have that are set up similarly and that have the same software (give or take). The others usually uppload something like 40-50 MB for an hour.

So, I’ve ran restic 0.12.0 with -vvvv and then grepd for MiB to find the lines that contains modifications that are in the size of MBs, and this is the result (having stripped out lines about com.apple.FinderInfo attributes, user/group and lines for files not having any extended attributes, to meet forum limitations):

$ ls -la@ Library/Containers
total 264
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.AboutThisMacLauncher
	com.apple.data-container-personality	1310155 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.AddressBook
	com.apple.data-container-personality	3236418 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.AddressBook.UrlForwarder
	com.apple.data-container-personality	474905 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.AirPlayUIAgent
	com.apple.data-container-personality	474894 
drwx------    4 <snip>     128 20 Feb 06:10 com.apple.Animoji.StickersApp.MessagesExtension
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.AppSSOKerberos.KerberosExtension
	com.apple.data-container-personality	971495 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.AppStore
	com.apple.data-container-personality	2526371 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.AppStoreDaemon.ASDAskPermissionExtension
	com.apple.data-container-personality	971020 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.AppleMediaServices.FollowUpExtension
	com.apple.data-container-personality	971024 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.AppleMediaServicesUI.SpyglassPurchases
	com.apple.data-container-personality	972305 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.AquaAppearanceHelper
	com.apple.data-container-personality	474522 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.AuthKitUI.AKAppSSOExtension
	com.apple.data-container-personality	972848 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.AuthKitUI.AKFollowUpServerUIExtension
	com.apple.data-container-personality	971022 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.CalendarFileHandler
	com.apple.data-container-personality	2925703 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.CalendarWidget.IntentsExtension
	com.apple.data-container-personality	971400 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.CaptiveNetworkAssistant
	com.apple.data-container-personality	476582 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Chess
	com.apple.data-container-personality	3080671 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.ClassKit.diagnosticextension
	com.apple.data-container-personality	971022 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.CloudDocs.MobileDocumentsFileProvider
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.CloudDocsDaemon.StorageManagement
	com.apple.data-container-personality	724413 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.CommerceKit.AskPermissionExtension
	com.apple.data-container-personality	971026 
drwx------@   4 <snip>     128 12 Feb 09:00 com.apple.CoreCDPUI.CDPFollowUpExtension
	com.apple.data-container-personality	971024 
drwx------    4 <snip>     128 12 Feb 08:52 com.apple.CoreRoutine.RTDiagnosticExtension
drwx------@   5 <snip>     160 12 Feb 08:52 com.apple.CryptoTokenKit.pivtoken
	com.apple.data-container-personality	974736 
drwx------@   5 <snip>     160 12 Feb 08:52 com.apple.CryptoTokenKit.setoken
	com.apple.data-container-personality	973924 
drwx------@   4 <snip>     128 12 Feb 09:00 com.apple.DiagnosticExtensions.sysdiagnose
	com.apple.data-container-personality	971013 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Dictionary
	com.apple.data-container-personality	2859942 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.DiscHelper
	com.apple.data-container-personality	475463 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.FaceTime
	com.apple.data-container-personality	2350643 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.FaceTime.IntentsExtension
	com.apple.data-container-personality	2350213 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Home
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   4 <snip>     128 12 Feb 08:57 com.apple.Home.HomeUIUserList
	com.apple.data-container-personality	2335881 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.IPAInstaller
	com.apple.data-container-personality	2462185 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.KerberosMenuExtra
	com.apple.data-container-personality	475814 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.MailCacheDelete
	com.apple.data-container-personality	2527569 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.MailShareExtension
	com.apple.data-container-personality	2528087 
drwxr-xr-x@   5 <snip>     160 12 Feb 08:58 com.apple.Maps
	com.apple.data-container-personality	1568960 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.MarkupUI.Markup
	com.apple.data-container-personality	971981 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.MarkupUI.MarkupPhotoExtension
	com.apple.data-container-personality	407179 
drwx------@   4 <snip>     128 12 Feb 08:58 com.apple.MobileSMS
	com.apple.data-container-personality	1813634 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Music.MusicCacheExtension
	com.apple.data-container-personality	971023 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Music.MusicStorageExtension
	com.apple.data-container-personality	372005 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.NetworkExtension.IKEv2Provider
	com.apple.data-container-personality	971019 
drwxr-xr-x@   5 <snip>     160 12 Feb 08:57 com.apple.Notes
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.Notes.IntentsExtension
	com.apple.data-container-personality	1818078 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Notes.QuickLookExtension
	com.apple.data-container-personality	1818087 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Notes.SharingExtension
	com.apple.data-container-personality	1818503 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Notes.SpotlightIndexExtension
	com.apple.data-container-personality	1817713 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.Notes.WidgetExtension
	com.apple.data-container-personality	1818065 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.OSDUIHelper
	com.apple.data-container-personality	474880 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.Pass-Viewer
	com.apple.data-container-personality	475859 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.PassKit.PaymentAuthorizationUIExtension
	com.apple.data-container-personality	972851 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.PhotoBooth
	com.apple.data-container-personality	732616 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.PhotoLibraryMigrationUtility
	com.apple.data-container-personality	474538 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.PhotoLibraryServices.PhotosDiagnostics
	com.apple.data-container-personality	971021 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Photos
	com.apple.data-container-personality	2448226 
drwx------@   4 <snip>     128 12 Feb 08:58 com.apple.Photos.PhotosFileProvider
	com.apple.data-container-personality	2447518 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.Photos.PhotosReliveWidget
	com.apple.data-container-personality	2447887 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Photos.PhotosSiriExtension
	com.apple.data-container-personality	2447519 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Photos.StorageManagementExtension
	com.apple.data-container-personality	2448163 
drwx------@   5 <snip>     160 12 Feb 08:52 com.apple.PressAndHold
	com.apple.data-container-personality	972500 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.Preview
	com.apple.data-container-personality	2699581 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.QuickTimePlayerX
	com.apple.data-container-personality	2665188 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.RAQLPreviewExtension
	com.apple.data-container-personality	971035 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.RAQLThumbnailExtension
	com.apple.data-container-personality	971037 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.RemoteManagement.AccountExtension
	com.apple.data-container-personality	971011 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.RemoteManagement.AppManagementExtension
	com.apple.data-container-personality	971017 
drwx------@   4 <snip>     128 12 Feb 09:00 com.apple.RemoteManagement.InteractiveLegacyProfilesExtension
	com.apple.data-container-personality	972646 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.RemoteManagement.LegacyProfilesExtension
	com.apple.data-container-personality	971018 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.RemoteManagement.ManagementTestExtension
	com.apple.data-container-personality	971018 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.RemoteManagement.PasscodeSettingsExtension
	com.apple.data-container-personality	972617 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMExtension.AppleInternal
	com.apple.data-container-personality	355659 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMExtension.Applications
	com.apple.data-container-personality	2205852 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMExtension.CloudFiles
	com.apple.data-container-personality	972373 
drwx------@   4 <snip>     128 12 Feb 09:00 com.apple.STMExtension.Developer
	com.apple.data-container-personality	2817612 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMExtension.GarageBand
	com.apple.data-container-personality	3784907 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.STMExtension.Mail
	com.apple.data-container-personality	2528797 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMExtension.OtherUsers
	com.apple.data-container-personality	335762 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMExtension.Trash
	com.apple.data-container-personality	2379454 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMExtension.iOSFiles
	com.apple.data-container-personality	1347144 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.STMFramework.UIHelper
	com.apple.data-container-personality	474880 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Safari
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Safari.CacheDeleteExtension
	com.apple.data-container-personality	1781624 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Safari.DiagnosticExtension
	com.apple.data-container-personality	1781629 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Safari.SafariQuickLookPreview
	com.apple.data-container-personality	1782006 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.SceneKitQLPreviewExtension
	com.apple.data-container-personality	971030 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.SceneKitQLThumbnailExtension
	com.apple.data-container-personality	971032 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.ScreenSaver.Computer-Name
	com.apple.data-container-personality	971957 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.ScreenSaver.Engine
	com.apple.data-container-personality	1773573 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.ScreenSaver.Engine.legacyScreenSaver
	com.apple.data-container-personality	971123 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.ScreenSaver.Engine.legacyScreenSaver.x86-64
	com.apple.data-container-personality	971130 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.ScreenSaver.iLife-Slideshow-Extension
	com.apple.data-container-personality	972784 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.ScreenSharing
	com.apple.data-container-personality	1470460 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.ScreenTime.ScreenTimeWebExtension
	com.apple.data-container-personality	971037 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.ScreenTimeWidgetApplication.ScreenTimeWidgetExtension
	com.apple.data-container-personality	419735 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.Siri
	com.apple.data-container-personality	3240597 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.Stickies
	com.apple.data-container-personality	728180 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.StorageManagementLauncher
	com.apple.data-container-personality	1310331 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.TV.TVCacheExtension
	com.apple.data-container-personality	971020 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.TV.TVStorageExtension
	com.apple.data-container-personality	2030049 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.TelephonyUtilities.PhoneIntentHandler
	com.apple.data-container-personality	971015 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.TextEdit
	com.apple.data-container-personality	2275218 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.VoiceMemos
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.WatchFaceAlert
	com.apple.data-container-personality	476191 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.WatchFaceAlert.ThumbnailExtension
	com.apple.data-container-personality	972091 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.archiveutility
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.calculator
	com.apple.data-container-personality	1188007 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.filevault
	com.apple.data-container-personality	971033 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.getmobilityinfo
	com.apple.data-container-personality	971039 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.installlog
	com.apple.data-container-personality	971036 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.maillogs
	com.apple.data-container-personality	971033 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.spotlight
	com.apple.data-container-personality	971046 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.syslog
	com.apple.data-container-personality	971031 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.systemprofile
	com.apple.data-container-personality	971039 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.timemachine
	com.apple.data-container-personality	971049 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.diagnosticextensions.osx.wifi
	com.apple.data-container-personality	971038 
drwx------@   4 <snip>     128 12 Feb 09:00 com.apple.family.FAFollowupExtension
	com.apple.data-container-personality	971028 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.finder.CreatePDFQuickAction
	com.apple.data-container-personality	972127 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.finder.MarkupQuickAction
	com.apple.data-container-personality	971984 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.finder.RotateQuickAction
	com.apple.data-container-personality	971804 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.finder.TrimQuickAction
	com.apple.data-container-personality	971840 
drwx------@   4 <snip>     128 12 Feb 08:57 com.apple.findmy
	com.apple.data-container-personality	1480207 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.findmy.FindMyNotificationsServiceExtension
	com.apple.data-container-personality	1479724 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.garageband10
	com.apple.data-container-personality	3783789 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.grapher
	com.apple.data-container-personality	3500456 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.iBooksX
	com.apple.data-container-personality	2027565 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.iBooksX.CacheDelete
	com.apple.data-container-personality	2026783 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.iBooksX.DiskSpaceEfficiency
	com.apple.data-container-personality	2027546 
drwx------@   5 <snip>     160 12 Feb 08:57 com.apple.iBooksX.SharingExtension
	com.apple.data-container-personality	2028387 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.iCal
	com.apple.data-container-personality	2921136 
drwx------@   4 <snip>     128 12 Feb 08:58 com.apple.iCal.CalendarNotificationContentExtention-OSX
	com.apple.data-container-personality	2920718 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.iCal.CalendarWidgetExtension
	com.apple.data-container-personality	2920690 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.iMovieApp
	com.apple.data-container-personality	1703184 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.iWork.Keynote
	com.apple.data-container-personality	815966 
drwx------@   5 <snip>     160 19 Apr 10:46 com.apple.iWork.Numbers
	com.apple.data-container-personality	1185718 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.iWork.Pages
	com.apple.data-container-personality	965068 
drwx------@   5 <snip>     160 12 Feb 08:52 com.apple.inputmethod.EmojiFunctionRowItem
	com.apple.data-container-personality	971582 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.mail
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   4 <snip>     128 12 Feb 08:58 com.apple.mail.MailNotificationContentExtension
	com.apple.data-container-personality	2527966 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.mail.MailQuickLookExtension
	com.apple.data-container-personality	2528091 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.mail.SpotlightIndexExtension
	com.apple.data-container-personality	2527581 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.mcx.ProfileHelper
	com.apple.data-container-personality	2325388 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.messages.AssistantExtension
	com.apple.data-container-personality	1812771 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.messages.StorageManagementExtension
	com.apple.data-container-personality	1813626 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.mobileslideshow.photo-picker
	com.apple.data-container-personality	2447878 
drwx------@   4 <snip>     128 12 Feb 08:58 com.apple.mobileslideshow.photospicker
	com.apple.data-container-personality	2447879 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.mobiletimer
	com.apple.data-container-personality	517620 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.mobiletimer-framework.MobileTimerIntents
	com.apple.data-container-personality	971407 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.mobiletimer.WorldClockWidget
	com.apple.data-container-personality	517226 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.notificationcenterui
	com.apple.data-container-personality	476017 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.osx-tailspin
	com.apple.data-container-personality	971031 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.podcasts
	com.apple.data-container-personality	2591698 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.podcasts.MacPodcastsStorageExtension
	com.apple.data-container-personality	2591690 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.podcasts.MacQuicklookExtension
	com.apple.data-container-personality	2592478 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.podcasts.PodcastsAuthAgent
	com.apple.data-container-personality	2592450 
drwx------@   4 <snip>     128 12 Feb 08:58 com.apple.podcasts.PodcastsNotificationExtension
	com.apple.data-container-personality	2592779 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.podcasts.SpotlightIndexExtension
	com.apple.data-container-personality	2591024 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.podcasts.widget
	com.apple.data-container-personality	2591667 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.preferencepane.security.AdvertisingExtension
	com.apple.data-container-personality	3826762 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.preferencepane.security.PrivacyAnalytics
	com.apple.data-container-personality	973062 
drwx------@   4 <snip>     128 12 Feb 09:00 com.apple.preferencepane.security.PrivacyTrackingAwareness.PrivacyPhotos
	com.apple.data-container-personality	971858 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.preferences.sharing.SharingBluetoothService
	com.apple.data-container-personality	972667 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.preferences.sharing.SharingPrefsExtension
	com.apple.data-container-personality	972498 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.quicklook.ui.helper
	com.apple.data-container-personality	474891 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.reminders
	com.apple.data-container-personality	1228415 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.reminders.WidgetExtension
	com.apple.data-container-personality	122841506 
drwx------@   5 <snip>     160 12 Feb 08:52 com.apple.reminders.macOSIntentsExtension
	com.apple.data-container-personality	1228974 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.reminders.quicklookextension
	com.apple.data-container-personality	1228407 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.reminders.sharingextension
	com.apple.data-container-personality	972061 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.replayKit.RPBroadcastActivityExtension-macOS
	com.apple.data-container-personality	971043 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.replayKit.RPVideoEditorExtension-macOS
	com.apple.data-container-personality	971037 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.share.AirDrop.send
	com.apple.data-container-personality	404863 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.share.Mail.compose
	com.apple.data-container-personality	3958687 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.share.Mail.compose-back-to-sender
	com.apple.data-container-personality	3958691 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.share.System.add-to-iphoto
	com.apple.data-container-personality	2449349 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.share.System.add-to-safari-reading-list
	com.apple.data-container-personality	973085 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.share.System.set-account-picture
	com.apple.data-container-personality	972886 
drwx------@   5 <snip>     160 12 Feb 09:00 com.apple.share.System.set-desktop-image
	com.apple.data-container-personality	973087 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.siriplugin.FileSearch
	com.apple.data-container-personality	1921814 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.ssinvitationagent
	com.apple.data-container-personality	3333136 
drwx------@   5 <snip>     160 12 Feb 08:58 com.apple.stocks
	com.apple.FinderInfo	    -1 
	com.apple.data-container-personality	    -1 
drwx------@   5 <snip>     160 12 Feb 08:59 com.apple.weather
	com.apple.data-container-personality	582544 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.weather.WeatherIntents
	com.apple.data-container-personality	582048 
drwx------@   4 <snip>     128 12 Feb 08:52 com.apple.weather.widget
	com.apple.data-container-personality	582047 
drwx------@   4 <snip>     128 12 Feb 08:59 com.apple.widgetkit.simulator
	com.apple.data-container-personality	2004296 
drwx------@   5 <snip>     160 17 Feb 09:34 com.microsoft.Excel
	com.apple.data-container-personality	382017 
drwx------@   5 <snip>     160 12 Feb 08:58 com.microsoft.Microsoft-Mashup-Container
	com.apple.data-container-personality	509608 
drwx------@   4 <snip>     128 12 Feb 08:58 com.microsoft.OneDrive.FileProvider
	com.apple.data-container-personality	719654 
drwx------@   5 <snip>     160 12 Feb 08:58 com.microsoft.OneDrive.FinderSync
	com.apple.data-container-personality	544606 
drwx------@   5 <snip>     160 12 Feb 08:58 com.microsoft.OneDriveLauncher
	com.apple.data-container-personality	544361 
drwx------@   5 <snip>     160 17 Mar 08:01 com.microsoft.Outlook
	com.apple.data-container-personality	843319 
drwx------@   4 <snip>     128 16 Mar 23:47 com.microsoft.Outlook.CalendarWidget
	com.apple.data-container-personality	842310 
drwx------@   5 <snip>     160 14 Apr 12:59 com.microsoft.Powerpoint
	com.apple.data-container-personality	508918 
drwx------@   5 <snip>     160 12 Feb 08:58 com.microsoft.SkyDriveLauncher
	com.apple.data-container-personality	544361 
drwx------@   5 <snip>     160 12 Feb 08:58 com.microsoft.SkypeForBusiness
	com.apple.data-container-personality	522555 
drwx------@   5 <snip>     160 24 Feb 11:05 com.microsoft.Word
	com.apple.data-container-personality	384048 
drwx------@   5 <snip>     160 12 Feb 08:59 com.microsoft.errorreporting
	com.apple.data-container-personality	1844789 
drwx------@   5 <snip>     160 12 Feb 08:59 com.microsoft.openxml.excel.app
	com.apple.data-container-personality	474531 
drwx------@   5 <snip>     160 24 Apr 19:24 com.microsoft.outlook.profilemanager
	com.apple.data-container-personality	2301771 
drwx------@   5 <snip>     160 14 Apr 20:37 com.microsoft.rdc.macos
	com.apple.data-container-personality	628029 
drwx------@   5 <snip>     160 12 Feb 08:58 com.microsoft.rdc.macos.qlx
	com.apple.data-container-personality	628020 
drwx------@   4 <snip>     128  9 Mar 10:45 net.sourceforge.grandperspectiv
	com.apple.data-container-personality	881062

Does anyone have any suggestions regarding what that extended attribute is and how to further narrow down what is causing restic to see this many MBs of changes in just a couple of minutes? Thanks!

punchcard · April 30, 2021, 3:29pm

My knowledge of Apple stuff is close to zero but this seems like it might be related to an Apple cloud based data storage. Perhaps the cloud storage app changes the file times to show that they have been synced with the cloud.
https://support.apple.com/en-ca/HT206985

Whether you decide to keep a local copy or not, a new Desktop and Documents folder is created in your home folder. If you choose to keep a local copy, your files in iCloud Drive are copied to a folder called iCloud Drive (Archive) in your home folder.

You’ve helped so many people in this forum. So hope this helps.

rawtaz · May 1, 2021, 4:34pm

@punchcard Thanks for chiming in Is there anything in particular that makes you suggest this in relation to the metadata on my folders under the ~/Library? I’m not seeing the connection.

One thing I should do is probably to investigate what that metadata actually is/contains, might give a clue as to why it changes.

punchcard · May 1, 2021, 7:12pm

The idea of metadata changes comes from a basic question of how can a user change so much data so quickly. To me they can’t type in that much data so there must be a program running which is changing the data. What I saw in the listing was the phrase “com.apple.data-container” and googled that. It seems the phrase is associated with the ICloud. From there is was just imagination of wondering how the os knows what has changed. And also wondering if the Mac has a directory being synced someone else’s machine through the ICloud and that other person is changing their files? Which in some ways is wondering are there might be or should be a business reason not to be backing up the changes coming from the cloud? All of this is me just going “I don’t know what the user is doing to generate so much data” and “Does the data fit within the responsibilities of your job to back it up?” Some of the items have the words “TV”, “podcasts”, “photos”, “quicktime” so the user might be listening to podcasts, tv etc which if not work related do not, or should not be backed up. These are all guesses.

punchcard · May 1, 2021, 7:14pm

The idea of metadata changes comes from a basic question of how can a user change so much data so quickly. To me they can’t type in that much data so there must be a program running which is changing the data. What I saw in the listing was the phrase “com.apple.data-container” and googled that. It seems the phrase is associated with the ICloud. From there is was just imagination of wondering how the os knows what has changed. And also wondering if the Mac has a directory being synced someone else’s machine through the ICloud and that other person is changing their files? Which in some ways is wondering are there might be or should be a business reason not to be backing up the changes coming from the cloud? All of this is me just going “I don’t know what the user is doing to generate so much data” and “Does the data fit within the responsibilities of your job to back it up?” Some of the items have the words “TV”, “podcasts”, “photos”, “quicktime” so the user might be listening to podcasts, tv etc which if not work related do not, or should not be backed up. These are all guesses.
This post and the previous one get posted but there is also a mysterious message box which shows up “500 internal server error” Don’t know anything about that either.

MichaelEischer · May 15, 2021, 4:06pm

If any of these folders changes then restic will have to reupload the whole directory. There’s currently no deduplication of large directory metadata.

The strange thing about the com.apple.data-container-personality extended attribute is that it seems to have shown up nowhere else on the entire internet. What data is contained in that extended attribute?

rawtaz · July 9, 2021, 4:00am

I’ll have to investigate that - haven’t had the time and a while ago the client took the laptop with her home over summer so I can’t easily access it anymore

MichaelEischer · February 5, 2022, 6:46pm

I’ve encountered the problem with the large metadata changes myself and did a bit of digging:

The problematic folder for me is also ~/Library/Containers and I’m also using Big Sur (just in version 11.6.2).

Navigating to the Containers folder using Finder shows two interesting things: firstly, several folders show custom icons and, secondly, the shown foldername is different from the name on the filesystem (-> the foldernames are translated). This matches with the observations from a closer look at the extended attribute:

It can be extracted using xattr -p com.apple.data-container-personality container.name.com | xxd -r -p > ~/container.name.com.plist. Afterwards either open it with Xcode or use defaults:

$ defaults read ~/container.name.com
{
    icon = {length = 895220, bytes = 0x69636e73 000da8f4 69633132 00000c80 ... 49454e44 ae426082 };
    name =     {
        Base = KeepingYouAwake;
        LSDefaultLocalizedValue = KeepingYouAwake;
[...]
    };
    pi = {length = 36, bytes = 0x00000000 12340000 08000000 12345678 ... 12340000 00000000 };
    version = 0;
}

That is essentially the metadata size problem is caused by the folder icons… I’ve deleted the extended attributes locally, but I’m not sure what the implications are, as there is absolutely no documentation on that attribute anywhere. The proper solution would be to allow restic to store large extended attributes just like normal data blobs.

MichaelEischer · February 5, 2022, 6:53pm

And now there’s an issue for that: https://github.com/restic/restic/issues/3643