Not at all. Start with the year you're in now. For earlier years, you can go back through photos, old school messages, report cards and anything else you already have and fill in only what you remember.
A memory book does not have to begin perfectly from day one to become meaningful later.