The first Fraser I have found is Joseph Fraser sr., probably born in the 1780s, place of origin unknown but quite likely to have been Scotland judging by his name. He was an engineer. His son Joseph Fraser jr., who was born in Liverpool about 1810, married Amelia Sally James in 1843 in London. They had four daughters and a son, Joseph Henry Fraser, all born in Aldgate. Both Joseph jr. and Joseph Henry were mariners in the merchant services. Joseph Henry was a lieutenant in the merchant navy and was almost certainly for a period in the 1870s sailing between England and New Zealand. During this period he fathered a daughter, my paternal grandmother, but it is very unlikely that he ever knew about this as she was unable to trace him. The trail goes cold as, being at sea, he would have missed the censuses and as yet I have not found any civil registration records for him apart from his birth. Joseph Henry's mother was alive and widowed in the 1881 census, living with one of her widowed daughters in Whitechapel. In the 1871 census Joseph had not yet gone to sea and was working as a carman. Joseph Fraser jr. was alive in the 1861 census but deceased before the 1871 census.
Update: I have found a Joseph Fraser in the 1901 census, right age and birthplace, working as a general labourer in the Southwark Union Infirmary. This man died in 1908 at the Camberwell Infirmary. It seems possible that Joseph returned to England some time between 1891 and 1901 in declining health (cardiac problems). He was not married and was described as homeless on his death certificate.