This　paper　proposes　a　medium　access　control　(MAC)　protocol　based　on　deep　reinforcement　learning　(DRL),　i.e.,　multi-channel　transmit　deep-reinforcement　learning　multi-channel　access　(MCT-DLMA)　in　heterogeneous　wireless　networks　(HetNets).　The　work　concerns　practical　unsaturated　channel　traffic　that　arrives　following　a　Poisson　distribution　instead　of　saturated　traffic　that　arrives　before.By　learning　the　access　mode　from　historical　information,　MCT-DLMA　can　well　fill　the　spectrum　holes　in　the　communication　of　existing　users.　In　particular,　it　enables　the　cognitive　user　to　multi-channel　transmit　at　a　time,　e.g.,　via　the　multi-carrier　technology.　Thus,　the　spectrum　resource　can　be　fully　utilized,　and　the　sum　throughput　of　the　HetNet　is　maximized.　Simulation　results　show　that　the　proposed　algorithm　provides　a　much　higher　throughput　than　the　conventional　schemes,　i.e.,　the　whittle　index　policy　and　the　DLMA　algorithms　for　both　the　saturated　and　unsaturated　traffic,　respectively.　In　addition,　it　also　achieves　a　near-optimal　result　in　dynamic　environments　with　changing　primary　users,　which　proves　the　enhanced　robustness　to　time-varying　communications.