In a large mixing bowl, whisk together the sweetened condensed milk, evaporated milk, and instant vanilla pudding mix until smooth. Let it rest for 5 minutes to thicken.
If using cream cheese, beat it until smooth and gently fold it into the pudding mixture.
Gently fold in the Cool Whip until light and fluffy.
In a 9x13-inch dish, layer vanilla wafers, banana slices, and pudding mixture. Repeat until all ingredients are used, ending with pudding on top.
Cover and refrigerate for at least 4 hours, or overnight for best results.